AFRL-RX- WP-TR-2  011-4176 


TECHNICAL  OPERATIONS  SUPPORT  III  (TOPS  III) 
Delivery  Order  0006:  An  Air  Force  Research  Laboratory  (AFRL)/RX 
Streamlined  Science  and  Technology  (S&T)  Planning  Guide  for 
Applying  Tailored  Systems  Engineering  (SE)  Guide  and  Companion 
Workbook 

Carol  Ventresca,  Bryan  DeHoff,  Thomas  Archer,  Robert  McCarty,  and  Stephanie  Globus 
SynGenics  Corporation 

Robert  Stroud 

R.B.  Stroud  &  Associates 
Gerald  Hasen 

Universal  Technology  Corporation 

Robert  Rapson,  James  Malas,  Robert  Enghauser,  and  William  Kesling 

Business  Operations  Branch 
Integration  &  Operations  Division 


JUNE  2011 
Final  Report 


Approved  for  public  release;  distribution  unlimited. 

See  additional  restrictions  described  on  inside  pages 


STINFO  COPY 


AIR  FORCE  RESEARCH  LABORATORY 
MATERIALS  AND  MANUFACTURING  DIRECTORATE 
WRIGHT-PATTERSON  AIR  FORCE  BASE,  OH  45433-7750 
AIR  FORCE  MATERIEL  COMMAND 
UNITED  STATES  AIR  FORCE 


NOTICE  AND  SIGNATURE  PAGE 


Using  Government  drawings,  specifications,  or  other  data  included  in  this  document  for  any 
purpose  other  than  Government  procurement  does  not  in  any  way  obligate  the  U.S.  Government. 
The  fact  that  the  Government  formulated  or  supplied  the  drawings,  specifications,  or  other  data 
does  not  license  the  holder  or  any  other  person  or  corporation;  or  convey  any  rights  or  permission  to 
manufacture,  use,  or  sell  any  patented  invention  that  may  relate  to  them. 

This  report  was  cleared  for  public  release  by  the  USAF  88th  Air  Base  Wing  (88  ABW)  Public 
Affairs  (AFRL/PA)  Office  and  is  available  to  the  general  public,  including  foreign  nationals. 

Copies  may  be  obtained  from  the  Defense  Technical  Information  Center  (DTIC) 

(http  ://www.  dtic  .mil) . 

AFRL-RX-WP-TR-201 1-4176  HAS  BEEN  REVIEWED  AND  IS  APPROVED  FOR 
PUBLICATION  IN  ACCORDANCE  WITH  THE  ASSIGNED  DISTRIBUTION  STATEMENT. 


*//Signature// 

ROBERT  L.  ENGHAUSER 
Tech  Business  Specialist 
Business  Operations  Branch 
Integration  &  Operations  Division 
Materials  and  Manufacturing  Directorate 


//Signature// 

KENNETH  A.  FEESER 
Chief,  Business  Operations  Branch 
Integration  &  Operations  Division 
Materials  and  Manufacturing  Directorate 


//Signature// 

ROBERT  L.  RAPSON 
Chief  Engineer 

Integration  &  Operations  Division 
Materials  and  Manufacturing  Directorate 


This  report  is  published  in  the  interest  of  scientific  and  technical  information  exchange,  and  its 
publication  does  not  constitute  the  Government’s  approval  or  disapproval  of  its  ideas  or  findings. 

*Disseminated  copies  will  show  “//Signature//”  stamped  or  typed  above  the  signature  blocks. 


REPORT  DOCUMENTATION  PAGE 


Form  Approved 
OMB  No.  0704-0188 


i 


The  public  reporting  burden  for  this  collection  of  information  is  estimated  to  average  1  hour  per  response,  including  the  time  for  reviewing  instructions,  searching  existing  data  sources,  gathering  and  maintaining 
the  data  needed,  and  completing  and  reviewing  the  collection  of  information.  Send  comments  regarding  this  burden  estimate  or  any  other  aspect  of  this  collection  of  information,  including  suggestions  for 
reducing  this  burden,  to  Department  of  Defense,  Washington  Headquarters  Services,  Directorate  for  Information  Operations  and  Reports  (0704-0188),  1215  Jefferson  Davis  Highway,  Suite  1204,  Arlington,  VA 
22202-4302.  Respondents  should  be  aware  that  notwithstanding  any  other  provision  of  law,  no  person  shall  be  subject  to  any  penalty  for  failing  to  comply  with  a  collection  of  information  if  it  does  not  display  a 

|  currently  valid  OMB  control  number.  PLEASE  DO  NOT  RETURN  YOUR  FORM  TO  THE  ABOVE  ADDRESS. 

1.  REPORT  DATE  (DD-MM-YY) 

2.  REPORT  TYPE 

3.  DATES  COVERED  (From  -  To) 

June  20 1 1 

Final 

14  June  2006  -  14  October  2010 

4.  TITLE  AND  SUBTITLE 

TECHNICAL  OPERATIONS  SUPPORT  III  (TOPS  III) 

Delivery  Order  0006:  An  Air  Force  Research  Laboratory  (AFRL)/RX  Streamlined 
Science  and  Technology  (S&T)  Planning  Guide  for  Applying  Tailored  Systems 
Engineering  (SE)  Guide  and  Companion  Workbook 

5a.  CONTRACT  NUMBER 

FA8650-06-D-5806-0006 

5b.  GRANT  NUMBER 

5c.  PROGRAM  ELEMENT  NUMBER 

62102F 

6.  AUTHOR(S) 

Carol  Ventresca,  Bryan  DeHoff,  Thomas  Archer,  Robert  McCarty, 
and  Stephanie  Globus  (SynGenics  Corporation) 

Robert  Stroud  (R.B.  Stroud  &  Associates) 

Gerald  Hasen  (Universal  Technology  Corporation) 

Robert  Rapson,  James  Malas,  Robert  Enghauser,  and  William  Kesling  (AFRL/RXOB) 

5d.  PROJECT  NUMBER 

4349 

5e.  TASK  NUMBER 

LO 

5f.  WORK  UNIT  NUMBER 

U2300012 

7.  PERFORMING  ORGANIZATION  NAM 

SynGenics  Corporation 

E(S)  AND  ADDRESS(ES) 

Business  Operations  Branch  (AFRL/RXOB) 

Integration  &  Operations  Division 

Air  Force  Research  Laboratory 

Materials  and  Manufacturing  Directorate 

Wright-Patterson  Air  Force  Base,  OH  45433-7750 

Air  Force  Materiel  Command,  United  States  Air  Force 

8.  PERFORMING  ORGANIZATION 
REPORT  NUMBER 

R.B.  Stroud  &  Associates 

Universal  Technology  Corporation 

9.  SPONSORING/MONITORING  AGENCY  NAME(S)  AND  ADDRESS(ES) 

Air  Force  Research  Laboratory 

Materials  and  Manufacturing  Directorate 

Wright-Patterson  Air  Force  Base,  OH  45433-7750 

Air  Force  Materiel  Command 

United  States  Air  Force 

10.  SPONSORING/MONITORING 
AGENCY  ACRONYM(S) 

AFRL/RXOB 

11.  SPONSORING/MONITORING 
AGENCY  REPORT  NUMBER(S) 

AFRL-RX-WP-TR-20 11-4176 

12.  DISTRIBUTION/AVAILABILITY  STATEMENT 

Approved  for  public  release;  distribution  unlimited. 


13.  SUPPLEMENTARY  NOTES 

PAO  Case  Number:  88ABW  2011-3570;  Clearance  Date:  20  Jun  2011.  Report  contains  color. 

Guide  and  companion  Workbook  can  be  separated  into  two  stand  alone  documents  allowing  the  workbook  worksheets  to 
serve  as  the  start  of  any  S&T  program  planning  activity  template  and  documentation. 

14.  ABSTRACT 

The  AFRL  Materials  and  Manufacturing  Directorate  (AFRL/RX)  Streamlined  S&T  Planning  Guide  and  companion 
Workbook  have  been  created  to  help  RX  apply  the  tenets  of  Systems  Engineering  (SE).  The  Guide  explains  how  to 
conduct  a  SE-based  S&T  planning  process  using  tailorable  worksheets  to  help  facilitate  the  process,  open 
communications,  and  capture  the  information  necessary  for  good  planning.  Although  focused  on  experience  in  Materials 
and  Manufacturing,  this  process  should  be  applicable  to  any  S&T  planning  process. 

The  Guide  presents  laboratory  planning  as  a  five-step  process;  1).  Form  Team  (Define  Problem),  2).  Identify 
Requirements,  3).  Generate  Alternatives,  4).  Evaluate  Alternatives,  and  5).  Develop/Deliver  Action  Plan.  The  activities 
cultivate  better  ideas,  involvement  and  communication. 

15.  SUBJECT  TERMS 

guide,  systems  engineering,  (SE),  laboratory,  planning,  process,  team,  problem,  requirement,  customer,  alternative, 
business  case  analysis,  value,  benefit,  develop,  deliver,  transition,  action,  plan,  materials,  manufacturing,  science, 
technology,  (S&T),  AFRL,  RX 


16.  SECURITY  CLASSIFICATION  OF: 

17.  LIMITATION 

18.  NUMBER 

19a.  NAME  OF  RESPONSIBLE  PERSON  (Monitor) 

a.  REPORT 

b.  ABSTRACT 

c.  THIS  PAGE 

OF  ABSTRACT: 

OF  PAGES 

Robert  L.  Enghauser 

Unclassified 

Unclassified 

Unclassified 

SAR 

126 

19b.  TELEPHONE  NUMBER  (Include  Area  Code) 

N/A 

Standard  Form  298  (Rev.  8-98) 

Prescribed  by  ANSI  Std.  Z39-18 


Table  of  Contents 


List  of  Figures . iii 

Foreword . iv 

Purpose . v 

Acknowledgments . vi 

Executive  Summary . 1 

Introduction  to  Systems  Engineering  Tailored  for  S&T  Planning . 3 

AFRL/RX  Streamlined  S&T  Planning  Process . 6 

STEP  1  -  FORM  TEAM . 9 

□  Worksheet  1.1  -  Define  Problem . 12 

□  Worksheet  1.2  -  Team  Make-up  /  Roles . 14 

□  Worksheet  1.3  -  Establish  Team  Charter . 16 

STEP  2  -  DETERMINE  REQUIREMENTS . 18 

□  Worksheet  2.1  -  List  of  Customer  Requirements . 23 

□  Worksheet  2.2  -  Technical  Challenge(s) . 25 

□  Worksheet  2.3  -  List  S&T  Exit  Criteria . 27 

□  Worksheet  2.4  -  Group  S&T  Exit  Criteria  by  Category . 29 

□  Worksheet  2.5  -  Group  S&T  Exit  Criteria  Complete  Description . 31 

STEP  3  -  GENERATE  ALTERNATIVES . 33 

□  Worksheet  3.1  -  List  Alternatives . 37 

□  Worksheet  3.2  -  Alternatives  Complete  Descriptions . 39 

STEP  4  -  EVALUATE  ALTERNATIVES . 41 

□  Worksheet  4.1  -  Desirability  for  each  S&T  Exit  Criteria  Vs.  each  Alternative . 48 

□  Worksheet  4.2  -  Risk  of  Achieving  each  S&T  Exit  Criterion  Vs.  each  Alternative . 50 

□  Worksheet  4.3  -  Composite  Scorecard . 52 

STEP  5  -  DELIVER  S&T  PLAN . 54 

□  Worksheet  5.1  -  Program  Action  Plan . 56 

□  Worksheet  5.2  -  SE  Case  Study . 58 


Approved  for  public  release. 


EXAMPLE  PROGRAM:  Aerospace  Ground  Equipment  Lifetime  Coating  Evaluation  for 
System  Sustainment  (AGELESS) . 60 

APPENDIX  SECTION . 76 

APPENDIX  A  -  Acronyms . 77 

APPENDIX  B  -  References . 78 

APPENDIX  C  -  Background  -  Systems  Engineering  (SE) . 79 

APPENDIX  D  -  Conducting  Brainstorming  Sessions . 86 

APPENDIX  E  -  Work  Breakdown  Structure  (WBS) . 89 

APPENDIX  F  -  Lessons  Learned  Regarding  SE  in  S&T . 90 

APPENDIX  G  -  Systems  Engineering  Guidance . 91 

APPENDIX  H  -  TRL  vs.  MRL . 93 


ii 

Approved  for  public  release. 


List  of  Figures 


Figure  ES  - 1.  SE  Tailored  for  S&T  Programs . 1 

Figure  ES  -  2.  Five-Step  Streamlined  S&T  Planning  Process  activities  and  outputs . 2 

Figure  1:  SE  Tailored  for  S&T  Programs . 3 

Figure  2:  Tailored  Systems  Engineering  for  S&T . 5 

Figure  3:  The  RX  5-Step  Streamlined  S&T  Planning  Process  and  Products . 6 

Figure  4.  The  Streamlined  SE  Process  and  accompanying  activity  Worksheets . 8 

Figure  5:  STEP  1  -  FORM  TEAM . 9 

Figure  6.  STEP  2  -  DETERMINE  REQUIREMENTS . 18 

Figure  7.  STEP  3  -  GENERATE  ALTERNATIVES . 33 

Figure  8.  Comparison  of  Alternative  Solutions . 35 

Figure  9:  STEP  4  -  EVALUATE  ALTERNATIVES . 41 

Figure  10:  STEP  5  -  DELIVER  S&T  PLAN . 54 

Figure  11.  Greatest  Trade  Space/Largest  Risk  Reduction  Opportunity  -  Exists  Between 

Milestones  (MS)  A  &  B . 82 

Figure  12.  8-  Key  Questions  presented  as  Systems  Engineering  'Vee' . 85 

Figure  13.  WBS  Example  to  functionally/systematically  breakdown  elements  of  the 

problem/environment . 89 

Figure  14.  Technology  Readiness  Levels  (TRL)  (Definitions  taken  from  DoD  5002) . 93 

Figure  15.  Manufacturing  Readiness  Levels  (MRL)  Definitions . 94 


iii 

Approved  for  public  release. 


Foreword 


The  ", Spirit "  of  Systems  Engineering 

The  Spirit  of  this  Guide  is  to  incorporate  Systems  Engineering  principles  into  any  planning  activity  and 
to  cultivate  an  atmosphere  of  open  "Conversation"  with  your  team  of  technical  experts  and  potential 
Customers/Stakeholders. 


"The  Power  of  System  Engineering  is  in  the  Discovery." 

The  Streamlined  Process  introduced  in  this  Guide  has  been  crafted  to  cultivate  discovery,  and  is  by 
design,  iterative  in  that  as  you  conduct  and  complete  each  sequential  activity,  you  and  your  team  may 
learn  new  insights  and  understandings  that  impact  and  build  upon  previous  findings. 


Incremental  SE  Rigor  -from  back  of  the  envelope  to  detailed  tool  assisted  analysis 

From  the  start  of  a  project,  document  "What  you  know  and  don’t  know,"  for  as  soon  as  an  idea  is 
written,  understanding  takes  place,  and  discussions  can  begin.  Conversation  is  Key!  With 
understanding  comes  comprehension,  and  the  better  ability  to  determine  the  amount  of  effort  (rigor) 
needed  to  invest  and  analyze  the  problem-solution  space.  From  a  small  team  or  Integrated  Product 
Team  ( IPT '),  to  full  up  modeling  and  simulation,  this  Guide  and  its  companion  Workbook  should  be 
usable  on  any  6.1,  6.2,  or  6.3  research  effort  using  an  incremental  "rigor"  approach. 


"Takes  Too  Long  and  Costs  Too  Much!" 

Many  may  feel  any  kind  of  Systems  Engineering  Process  takes  too  long  and  costs  too  much;  however,  if 
the  right  discussions  and  decisions  are  made  upfront  thus  avoiding  significant  errors ,  then  the 
investment  of  time  is  usually  worth  the  effort. 


The  Pirates’  Code  -  "...more  what  you’d  call  "Guidelines"  than  actual  rules?" 

( Pirates  of  the  Caribbean:  The  Curse  of  the  Black  Pearl  (2003 ) 

In  the  spirit  of  the  code,  the  Process  Steps  in  this  Guide  are  but  "Guidelines  and  not  actual  rules” 
....meaning,  nothing  in  this  Guide  is  to  be  viewed  as  required,  in  fact,  everything  can  be  tailored,  and 
customization  is  highly  encouraged  to  fit  your  situation.  The  goal  is  to  create  an  environment  to 
generate  and  document  the  best  discussions  (...The  SE  Conversation). 

Robert  Rapson 
Chief  Engineer 

Materials  and  Manufacturing  Directorate 
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Purpose 


The  AFRL  Materials  and  Manufacturing  Directorate  (AFRL/RX]  Streamlined  S&T  Planning 
Guide  and  Workbook  have  been  created  to  help  RX  apply  the  tenets  of  Systems  Engineering  (SE], 
The  Guide  explains  how  to  conduct  a  SE-based  S&T  planning  process  using  tailorable  worksheets  to 
help  facilitate  the  process,  open  communications,  and  capture  the  information  necessary  for  good 
planning.  SE  Facilitator  support  is  available  from  the  AFRL/RX  Systems  Engineering  Working 
Group  (SEWG]  for  any  RX  Program  Manager  wanting  to  use  this  Guide  on  their  project. 

Although  focused  on  experience  in  Materials  and  Manufacturing,  the  process  should  be 
applicable  to  any  S&T  planning  process. 

Using  the  Guide  and  Workbook  should  enable  an  S&T  Team  to  create  and  document  a  SE- 
based  program  plan  using  a  "customer  focused"  tailored  SE  approach. 

AFRL/RX  is  pursuing  this  tailored  approach  to  SE  in  order  to: 

•  Comply  with  Department  of  Defense,  Air  Force,  and  Air  Force  Research 
Laboratory  guidance. 

•  Develop  an  SE  Culture  appropriate  for  an  S&T  environment 

•  Improve  Program  Management  effectiveness  and  efficiency 

•  Improve  the  quality  and  success  of  technology  transitions  and  deliveries. 

Why  is  strategic  program/project  planning  with  a  tailored  SE  process  important? 

•  Help  understand  where  technology  /  concept  ideas  fit  customer  requirements 

•  Provide  a  solid  basis  for  approval  /  maintenance  of  funding 
(marketing/ advocacy] 

•  Build  a  firm  foundation  for  the  increased  likelihood  of  program  success 

•  Link  AF  S&T  Vision  to  AFRL  products 
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Take  time  and  think  well  upon  your  subject. 
Nothing  valuable  can  be  lost  by  taking  time. 

Abe  Lincoln 


A  Systems  Engineering  approach  allows  a  methodical  process 
to  explore  and  select  the  optimized  actions 
to  plan  and  execute  better  S&T programs. 
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Executive  Summary 


RX  is  implementing  Systems  Engineering  (SE)  in  accordance  with  guidance  from  the 
Department  of  Defense  and  Air  Force  policies. 

RX  SE,  as  adapted  for  Science  &  Technology  (S&T),  has  its  roots  in  the  AFRL 
Affordability  Initiative,  the  essence  of  which  has  been  vetted  in  the  Integrated  Product  & 
Process  Development  (IPPD)  SE  management  approach. 

SE  tailored  for  S&T  consists  of  three  major  phases,  1).  Plan,  2).  Execute,  and  3). 
Deliver  /  Transition  technology  capability  (Figure  ES  -  1).  This  Guide  provides  an  outline 
for  Phase  #1  Plan  (Figure  ES-2).  For  phase  #2  Execute,  the  SE  principles  are  applied  when 
reporting  the  answers  to  the  8-Key  Questions  (SE  "Vee")  at  AFRL/RX  Lab  Management 
Reviews  (LMRs)  and  Program  Baseline  Reviews  (PBRs). 


5-Step  Streamlined 
Planning 


"Plan  the  Program  Right" 


8-Key  Question 
Assessment 


'Consistent  SE  Assessment  ...6.1,  6.2,  6.3,  ATD" 


S&T  to 
External 
Customers 


HE  HE1  El 


1.  Form  Team 

2.  Determine  Requirements 

3.  Generate  Alternatives 

4.  Evaluate  Alternatives 

5.  Deliver  S&T  Plan 


SE  "Vee" 


1.  Customer 
2.  Requirements 
3.  Demonstration 
4.  Tech  Options 


Often  an  Iterative  Process 


8.  Transition  Plan 
7.  Program  Structure 
6.  Risks 

5.  Best  Approach 


S&T  to 
Internal 
Lab 

Customers 


Figure  ES  - 1.  SE  Tailored  for  S&T  Programs 

The  RX  SE  team  developed  a  five-step  Streamlined  S&T  Planning  Process  to  address 
the  two  challenges  perceived  in  S&T  -  " it  costs  too  much  and  it  takes  too  long.”  This 
planning  process  adapts  the  basic  SE  principles  in  IPPD  and  other  techniques  for  an  SE- 
based  approach  for  programs  at  any  level  of  S&T  maturity. 
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Figure  ES  -  2  lays  out  the  basic  Streamlined  S&T  Planning  Process  to  conducting  the 
initial  and  any  subsequent  project  planning,  at  minimal  time  and  expense.  References 
throughout  the  Guide  point  to  other  SE  tools  if  a  more  in-depth  analysis  is  required. 


The  colors  in  the  AFRL/RX  Streamlined  S&T  Planning  Process  have  a  specific 
meaning: 


•  Blue  -  denotes  "Problem"  Space  -  identifying  the  key  program 
characteristics,  which  includes  performance  parameters,  etc. 

•  Yellow  -  denotes  "Solutions”  or  Trade  Space  -  Alternative  Analysis 

•  Green  -  shows  desired  documentation  of  the  planning  process 


Step  1  Step  2  Step  3  Step  4  Step  S 


Form 
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Deliver 

Team 

Requirements 
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Stakeholders 

•  Establish  Planning 

Team 

•  Define  Requirements 

•  DefmeTechnology 

Challenges 

•  Define  S&T 

Ex#  Critena 

•  Validate  wf  Customer 

•  Understand  Relevant 
State-of-the-Art 

Technologies 

•  Generate  Altematrve 

Solutions 

•  Assess  Alternatives 

-  Score  S&T 

Ex#  Cntena 

-  Evaluate  Risk 

•  Select  Preferred 

Altematrve(s) 

•  Finalize  AF  Problem 
/  Goals  /  Preferred 

Solubon(s) 

•  Prepare  for  Intended 

Action  Course 

Document: 

□  Problem  Definition 

□  Team  Roster 
(includes  roles  & 

responsibilities) 

□  Team  Charter 

□  Requirement  Set 

-  Performance 

-  Affordability 

-  ProduciMity 

-  Supportability 

-  Others 

□  Tech  Chalenge  List 

□  S&T  Ex#  Critena  Set 

□  Full  Descnption  of 
each  Altematrve, 
including 

-  TRL  Levels 

-  MRL  Levels 

-  Schedule  /  Key 

Milestones 

-  Cost  Estimate 

□  Value  Analysis 

-  Alternative  Scores 
-Top  Fhsks& 
Mitigation  Plans 

□  Rationale  for 
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Altematrve(s) 

□  AF  Problem  1  Goal 
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-  Roadmap 

□  Action  Plan 

Ba&edonS&T 


Figure  ES  -  2.  Five-Step  Streamlined  S&T  Planning  Process  activities  and  outputs 
The  RX  planning  method  forms  the  foundation  of  the  SE  Conversation  and  include: 


•  The  Planning  Process  must  include  a  team  of  knowledgeable  stakeholders 

•  The  S&T  Planning  Process  is  highly  collaborative  and  iterative  in  nature 

•  The  level  of  rigor  required  is  tailorable  to  the  specific  S&T  problem 

•  Documentation  is  required  at  each  step  of  the  process 
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Introduction  to  Systems  Engineering  Tailored  for  S&T  Planning 


RX  SE,  as  tailored  for  S&T,  has  its  roots  in  the  AFRL  Affordability  Initiative,  the 
essence  of  which  has  been  vetted  in  the  Integrated  Product  &  Process  Development  (IPPD) 
SE  management  approach.  Recent  positive  experience  with  tailored  SE  in  AFRL  S&T  has 
demonstrated  the  relevance  and  benefit  for  AFRL  programs. 


5-Step  Streamlined 
Planning 


“Plan  the  Program  Right" 


8-Key  Question 
Assessment 


'Consistent  SE  Assessment  ...6.1,  6.2,  6.3,  ATD" 


S&T  to 
External 
Customers 


fi  fi  fi  fi  fi 


1.  Form  Team 

2.  Determine  Requirements 

3.  Generate  Alternatives 

4.  Evaluate  Alternatives 

5.  Deliver  S&T  Plan 


SE  "Vee" 


1.  Customer 
2.  Requirements 
3.  Demonstration 
4.  Tech  Options 


Often  an  Iterative  Process 


8.  Transition  Plan 
7.  Program  Structure 
6.  Risks 

5.  Best  Approach 


S&T  to 
Internal 
Lab 

Customers 


Figure  1:  SE  Tailored  for  S&T  Programs 

For  any  existing  or  proposed  S&T  program,  tailored  SE  consists  of  three  phases,  as 
shown  in  Figure  1: 

•  In  the  five-step  streamlined  planning  phase,  tailored  SE  principles  are  used  to 
carefully  craft  an  S&T  program  that  is  executable  and  meets  customer 
expectations.  Steps  in  this  planning  phase,  along  with  tools  available  for  use  in 
this  phase,  will  be  discussed  in  detail  in  this  Guide. 

•  In  the  program  execution  phase,  the  eight  SE  questions  shown  in  Appendix  C 
are  regularly  used  to  ensure  each  S&T  program  continues  to  progress  in  a 
satisfactory  manner.  AFRL/RX  Instruction  61-104  addresses  this  phase. 

•  In  the  last  phase,  the  delivery/transition  or  close  out  phase,  the  technology 
products  matured  by  an  S&T  program  and  meeting  the  customer  expectations 
called  out  in  the  planning  phase,  are  either  delivered  to  another  internal  S&T 
customer  (6.1  technologies  delivered  to  a  corresponding  6.2  Applied  Research 
program,  or  6.3  technology  products  delivered  to  another  Technology 
Directorate’s  6.3  integration  program);  or  transitioned  to  an  external  AF  or  DoD 
customer  for  insertion  into  a  military  system. 
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This  three-phase  process  is  often  an  inherently  iterative  activity  in  an  S&T 
environment,  as  plans  are  adjusted  with  maturing  technology  and  evolving  customer 
requirements  and  expectations.  Additionally,  many  S&T  programs  will  deliver 
technologies  in  a  spiral  manner,  with  each  spiral  providing  greater  depth  in  the  form  of 
more  improved,  capable  and  affordable  product. 

Although  the  word  “transition"  is  sometimes  used  to  depict  the  internal  AFRL 
delivery  of  S&T  products,  the  DoD/DAU  Manager’s  Guide  to  Technology  Transition  in  an 
Evolutionary  Acquisition  Environment  definition  of  transition,  tailored  for  Air  Force 
applications,  is  defined  as  follows: 

•  Technology  transition  is  the  use  of  technology  in  military  systems  to  create 
effective  weapons  &  support  systems 

•  Technology  transitions  can  occur  during  the  development  of  new  systems,  or 
after  a  system  has  been  fielded  for  a  number  of  years 

•  Technology  transitions  can  occur  between  government  organizations,  such  as 
when  AFRL  transitions  a  technology  to  an  AFMC  Product  Center  for  use  in  a 
specific  AF  system;  or  between  government  and  industry,  such  as  when  AFRL 
transitions  technology  to  a  System  prime  or  sub-contractor;  and  vice  versa,  such 
as  when  industry  transitions  technology  to  an  AFMC  Product  Center. 


A  broad  spectrum  of  SE-based  methods  and  toolsets  are  available  commercially. 
Decisions  regarding  which  method,  from  relatively  quick  and  simple,  to  complex  and  more 
time-consuming,  might  prove  cost  effective  in  any  given  case  depends  primarily  on  the 
level  of  technology  maturity  and  particular  end-application  involved.  It  is  incumbent  on 
the  S&T  program  manager  to  determine  the  level  of  SE  analysis  most  appropriate  to 
support  decisions  required  for  a  particular  S&T  program.  The  RX  SEWG  (The  SE  Team)  can 
assist  in  determining  the  tools  most  appropriate  for  the  problem  at  hand. 

In  tailoring  SE  for  application  to  S&T,  it  is  useful  to  think  of  the  range  of  available  SE- 
based  methods  as  a  continuum,  as  illustrated  in  Figure  2.  In  this  figure,  the  vertical  axis 
depicts  the  rigor,  also  highly  related  to  the  complexity,  of  SE  planning  and  analysis  methods 
ranging  from  simply  asking  the  eight  SE  questions  to  pursuing  a  full-blown  modeling  and 
simulation  analysis  at  the  top  of  the  scale.  For  many  basic  and  applied  research  S&T 
efforts,  the  eight-question  level  of  conversation  will  suffice,  whereas  for  6. 2/6. 3  programs 
delivering  technology  products  to  a  customer,  much  greater  qualitative  and  quantitative 
levels  of  analysis  may  be  required  to  ensure  customer  acceptance  of  the  technology 
deliverables. 
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Tailored  SE  Toolbox  for  S&T  Program  Planning 


High 


Level  of 
SE  Rigor 


Low 


•  Quantitative  SE 
w /  MS&A 

•  Quantitative  SE 
Trade  Analysis 

•  More  Rigorous 
Quantitative  SE 


•  Streamlined 
Quantitative  SE 

•  Web-Based 
T  raining/References 

•  8  SE  Questions 


5 -Step  Streamlined 
SE  Process: 

-  Light-Medium  Rigor 
-6.1  to  6.3 


The  "Conversation 


// 


6-1  6.2  6.3  ATD 

Level  of  Tech  Maturity/Integration  Required 

Figure  2:  Tailored  Systems  Engineering  for  S&T 


Tailored  SE  can  be  accomplished  effectively  in  RX  by  applying  the 
approach  that  best  fits  each  case;  owned  by  the  team,  appropriate  to  the 
requirement,  and  using  analytical  tools  as  appropriate 
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AFRL/RX  Streamlined  S&T  Planning  Process 


The  RX  Streamlined  S&T  Planning  Process,  as  shown  in  Figure  3,  is  not  a  new 
concept,  but  is  founded  on  the  Integrated  Product  and  Process  Development  (IPPD) 
process  that  emerged  in  AFRL  from  the  Affordability  Initiative  of  the  1990s. 

The  value  of  the  streamlined  process  is  two-fold.  First,  a  quick  and  focused 
application  of  this  planning  process  can  reveal  the  right  technology  direction  for  the  S&T 
team  to  pursue.  Second,  following  the  process  can  foster  the  right  choices  regarding  what 
level  of  SE  analysis  would  ultimately  be  appropriate  for  the  case  at  hand. 


Step  1  Step  2  Step  3  Step  4  Step  5 


Do: 

•  Oefne  Problem 

•  Identify  All 

Stakeholders 

•  Establish  Planning 

Team 

•  Define  Requirements 

•  Define  Technology 

Challenges 

•  Define  S&T 

ExitCritena 

•  Validate  Customer 

•  Understand  Relevant 
State-of-th  e-Art 

Technologies 

•  Generate  Altematrve 

Solutions 

•  Assess  Alternatives 

-  Score  S&T 

Exit  Cntena 

-  Evaluate  Risk 

•  Select  Preferred 

Altematrve(s) 

•  Finalize  AF  Problem 
/  Goals  /  Preferred 

Solubon(s) 

•  Prepare  for  Intended 

Action  Course 

Document: 

□  Problem  Definition 

□  Team  Roster 
(includes  roles  & 

responsibilities) 

□  Team  Charter 

□  Requirement  Set 

-  Performance 

-  Affordability 

-  Producibility 

•  Supportability 

•  Others 

□  Tech  ChaBenge  List 

□  S&T  Exit  Cntena  Set 

□  Full  Descnpdon  of 
each  Alternative, 
including 

-  TRL  Levels 

-  MRL  Levels 

-  Schedule  /  Key 

Milestones 

-  Cost  Estimate 

□  Value  Analysis 

-  Alternative  Scores 
-Top  Risks  & 
Mitigation  Plans 

□  Rationale  for 

Preferred 

Altematrve(s) 

□  AF  Problem !  Goal 

□  Preferred  Solubon(s) 

-  Roadmap 

□  Action  Plan 

Bk*I«i5S.T  IPfDPl««tt*'.«5lc«3-»02l 


Figure  3:  The  RX  5-Step  Streamlined  S&T  Planning  Process  and  Products 


The  five-Step  process  is  not  serial.  Iteration  happens  when  understanding, 
requirements,  and  customers  change,  or  when  new  technology  solutions  become  viable. 
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Based  on  experience  of  applying  this  process,  there  are  two  secrets  to  success: 


First,  total  commitment  -  The  first  step  at  the  beginning  of  this  process  is  getting 
commitment  between  the  requestor  of  the  effort  and  the  members  of  the 
organization  using  the  process.  Without  a  formal  commitment  including 
accountability,  the  process  drags  out  and  becomes  frustrating  to  all  involved. 

Second,  complete  documentation  of  each  step  is  critical  for  success. 

By  the  end  of  this  SE  process,  the  anticipated  products  include: 

•  Problem  Definition 

•  Team  Listing 

•  Prioritized  Requirements 

•  S&T  Exit  Criteria 

•  Alternative  Solutions 

•  Evaluation  of  Alternatives,  including  selection  of  proposed  program  approach 

•  A  documented  S&T  Plan  built  from  the  above  activities. 


The  worksheets  provided  in  the  Guide  and  Workbook  are  not  intended  to  be  the 
"end-all"  list  of  questions.  They  are  simply  a  starting  point  to  assist  the  team  to  ensure  all 
appropriate  topics  are  considered  and  to  guide  the  team  through  the  planning  process. 

The  Worksheets  available  in  this  Guide  and  in  the  companion  Workbook  are 
provided  to  foster  a  TEAM  approach,  to  decompose  the  problem  and  to  stimulate  creativity 
and  discussions  by  using  the  principles  of  SE  to  analyze  the  solutions. 

The  following  diagram  provides  an  overview  of  the  Streamlined  SE  Process  along  with  the 
activity  Worksheets  created  to  generate  discussion  and  provide  a  starting  point  for 
documenting  each  step.  Detailed  instructions  for  each  Worksheet  are  available  in  this 
Streamlined  S&T  Planning  Guide. 


7 

Approved  for  public  release. 


Problem  Space 


Solution  Space 


Step  1 


Step  2 


Step  3 


Step  4 


Step  5 


FORM 

TEAM 


DETERMINE 

REQUIREMENTS 


GENERATE 

ALTERNATIVES 


EVALUATE 

ALTERNATIVES 


DELIVER 
S&T  PLAN 


Figure  4.  The  Streamlined  SE  Process  and  accompanying  activity  Worksheets 


The  Worksheets  for  the  above  activities  can  be  tailored,  expanded  upon, 
customized  or  ignored  to  suit  the  needs  of  your  planning. 


So,  Let's  Get  Started! 
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STEP  1  -  FORM  TEAM 


Step  1 


Do: 

•  Defre  Problem 

•  Identify  All 

Stakeholders 

•  Establish  Planning 

Team 


Document: 

□  Problem  Definition 

□  Team  Roster 
(includes  roles  & 

responsibilities) 

□  Team  Charter 


Step  2 


Step  3 


Step  4 


□  f 


a1 
□  ; 


a 


□  Ratn 


Step  5 


Figure  5:  STEP  1  -  FORM  TEAM 


Step  1  of  the  Planning  Process  involves  understanding  (defining)  the  real  problem 
and  the  formation  of  the  right  team  of  stakeholders.  Whether  a  small,  two  or  three 
member  team,  or  a  larger  Integrated  Product  Team  (IPT),  working  the  steps  of  the  process 
in  this  Guide  results  in  improved  communication. 

Note:  the  Problem  Space  addressed  in  S&T  planning  can  vary  greatly. 

•  In  Basic  and  Applied  Research,  the  Problem  Space  may  be  very  general, 
advancing  the  science  in  a  given  technical  area  with  a  range  of  possible 
engineering  applications. 

•  For  an  Advanced  Technology  Development  program,  there  is  usually  a  much 
more  specific  technology  pull,  where  a  Warfighter  customer  needs  a  material 
solution  to  a  specific  capability  gap. 
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This  SE  Guide  can  be  applied  with  equal  success  across  the  entire  S&T  spectrum, 
from  low  TRL  S&T  programs  to  those  involving  transition  to  specific  problem  owners  and 
end  customers.  In  this  regard,  the  team  has  to  be  flexible  in  how  it  approaches  each 
planning  task.  It  may  be  equally  productive  to  focus  on  a  specific  transition  directly  to  a 
customer  or  on  a  low  TRL  challenge,  which  may  transition  through  another  research  group, 
even  another  Technology  Directorate  (TD),  for  necessary  technology  maturation  and 
subsequent  integration  into  a  specific  system. 

The  key  point  is  the  team  can  use  the  logic  flow  in  this  streamlined  SE  method  to 
plan  at  whatever  level  of  technology  maturity  inherent  in  the  S&T  challenge  at  hand. 
Flexibility  and  commitment  are  required  for  the  team  to  develop  a  tailored  program  plan 
that  they  own. 

To  be  effective,  the  S&T  planning  team  needs  the  right  people: 

•  Team  Leader:  Typically,  the  Program  or  Project  Manager  (PM)  is  required  to 
lead  the  team  and  interface  with  customers. 

•  Team  members:  carefully  selected  to  fit  the  project  challenge,  committed  to  the 
process  by  participating  in  team  meetings,  accomplishing  the  tasks  required  by 
the  process,  including  homework  between  meetings  and  drawing  on  subject 
matter  expertise  when  required. 

•  Scribe:  a  dedicated  member  recording  the  meeting  discussions.  The  role  played 
by  the  scribe  needs  to  be  both  reflective  as  well  as  forward  looking.  The  scribe 
cannot  effectively  serve  as  a  meeting  participant  as  well.  The  RX  SE  Office  can 
assist  with  this  role  if  necessary. 

•  SE  Facilitator:  can  be  made  available  in  RX  to  coach  the  S&T  team  members 
with  regard  to  the  process,  facilitate  meetings,  and  ensure  that  documentation 
critical  to  success  is  effectively  captured  and  maintained  by  the  team.  A 
Facilitator  is  not  required  and  teams  can  use  this  process  autonomously;  a 
Facilitator  is  recommended  if  the  team  has  no  prior  experience  in  SE. 

For  Step  1,  the  Team  Leader  could  conduct  a  Team  Orientation  Meeting  to  review 
the  expected  roles  and  responsibilities  of  the  team  members  and  ensure  each  member 
understands  expectations.  The  PM  or  an  SE  Facilitator  can  present  the  RX  Streamlined  S&T 
Planning  Process  and  a  Project  Overview  by  reviewing  what  is  known  of  the  AF  Problem 
Statement  and  the  results  of  any  preliminary  problem  exploration  by  the  PM. 

A  beneficial  team  exercise  at  this  early  stage  of  problem  definition  is  to  construct  a 
"Systems"  functional  work  breakdown  structure  of  the  problem  and  the  environment  in 
which  the  problem  exists,  identifying  the  inputs  and  outputs,  the  complexity  and 
relationships  between  functional  systems.  More  information  concerning  diagramming  a 
work  or  functional  breakdown  structure  can  be  found  at  Appendix  E. 
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Step  1 


Form 

Team 


•  Define  Problem 

•  Identify  All 

Stakeholders 

•  Establish  Planning 

Team 


□  Problem  Definition 

□  Team  Roster 
(includes  roles  & 

responsibilities) 

□  Team  Charter 


Step  1  -  FORM  TEAM 

(See  Streamlined  SE  Process  Workbook  for  suggested  Worksheets) 


□  Worksheet  1.1  -  Define  Problem 

□  Worksheet  1.2  -  Team  Make-up  /  Roles 

□  Worksheet  1.3  -  Establish  Team  Charter 


Pre  -  Meeting  Homework: 

The  S&T  PM  and  or  a  small  team  of  close  Tech  Advisors 
should  accomplish  the  three  Step  1  worksheets  individually 
as  a  first  cut,  before  requesting  the  support  of  a  larger  team. 


Three  worksheets  are  part  of  Step  1  to  help  identify  and  understand  the  problem 
space  and  build  an  effective  team.  All  worksheets  are  also  available  in  the  companion 
Workbook  to  this  Guide.  The  Worksheets  are  designed  to  spur  creative  thoughts  and 
discussion  and  should  take  no  more  than  10-20  minutes  each  to  complete. 


"If you  can’t  explain  it  simply,  you  don’t  understand  it  well  enough."  Albert.  Einstein 


"We  can’t  solve  problems  by  using  the  same  kind  of  thinking  we  used  when  we  created  them.” 

-Albert.  Einstein 
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□  Worksheet  1.1  -  Define  Problem 


Who  is  the  Customer?  ( Who  brought  the  problem?) 

In  technology  programs  there  may  be  multiple  organizations  who  will  use  the 
technology  (particularly  for  lower  TRL  programs).  The  first  test  in  identifying  a  customer 
is  the  financial  test.  Customer  organizations  usually  have  a  direct  financial  investment  in 
continuing  the  technology  development/maturation  at  the  conclusion  of  the  program 
under  consideration.  Externally  funded  programs  usually  have  external  customers. 

Time  frame  of  Problem  [When/Urgency  of  Need?)-. 

The  first  data  point  to  identify  is  the  time  frame  associated  with  a  customer’s  need. 
Is  it  a  near-,  mid-,  or  far-term  problem?  This  can  be  an  important  constraint  to  the 
proposed  problem  response. 

Describe  the  Problem  [Scope/Major  Issues/Constraints)-. 

A  good  problem  description  captures  the  primary  issue  and  any  supporting  facts  in 
terms  that  relate  to  the  Air  Force  or  the  DoD  need.  For  example,  the  root  cause  might  be 
scoped  as  a  performance,  affordability,  or  reliability  issue,  or  other  as  determined. 

Who  Owns  the  Problem?  (Who  has  the  task  to  solve  the  problem?) 

In  this  process,  the  Problem  Owner  is  the  organization  responsible  for 
implementing  the  identified  material  solution  to  the  problem.  Air  Force  problems  are 
usually  owned  by  either  acquisition  or  sustainment  organizations,  or  even  the  Warfighters 
themselves  (including  field  maintainers).  Members  of  the  AFRL  community  working 
through  this  process  are  typically  not  the  Problem  Owner,  but  rather  the  solution  owner. 

Who  is  the  End  User?  (Who  turns  the  wrench/pushes  the  button?) 

Most  of  the  time  the  end  user  is  not  the  problem  owner  .  In  most  cases,  the  end  user 
is  the  operational  user  of  the  Air  Force  system  incorporating  the  technology  solution. 

Why  Should  AFRL/RX  be  Working  This  Issue? 

How  does  this  problem  fit  within  AFRL/RX?  Is  there  a  clear  alignment  with  a  Core 
Technical  Competency,  or  established  product  line?  Are  there  other  AFRL  Directorates  or 
outside  organizations  working  this  issue  as  well? 

Who  Needs  to  be  Included  as  Partners  in  this  Issue? 

The  Team  needs  to  consider  if  other  RX  Divisions,  AFRL  Directorates,  or  external 
organizations  should  be  included  in  this  effort. 

What  Do  We  NOT  Know? 

As  part  of  defining  the  problem  space,  capturing  aspects  of  the  problem  that  are 
unknown  or  unspecified  will  help  guide  subsequent  discussions  by  the  team. 
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Step  1:  Form  Team.  Start  to  think  of  the  team  in  specific  terms  of  the  problem  at  hand. 
Different  problems  require  different  member  experiences  and  skills. 


Worksheet  1.1 
Define  Problem 

The  team  works  with  the  Customer(s)/StakehoIder(s )  to  scope  the  problem  space. 
Who  is  the  Customer?  (Who  brought  the  problem?) _ 


Project  Name 
Member  Name 
Role 

Worksheet  Date 


Time  frame  of  Problem  (When  Needed/Urgency?) _ 

Describe  the  Problem  (Scope/Major  Issues/Constraints) 


Who  Owns  the  Problem?  (Who  has  the  task  to  solve  the  problem ?) 


Who  is  the  End  User?  (Who  turns  the  wrench/pushes  the  button ?) 


Why  should  AFRL/RX  be  working  this  issue? 


Who  Needs  to  be  included  as  Partners  in  this  issue? 


What  Do  We  NOT  Know? 


...  the  "SE  Conversation"  begins  with  understanding  of  the  problem.. .What  is  the  "Discontent?"  ...and  Why? 
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□  Worksheet  1.2  -  Team  Make-up  /  Roles 

Worksheet  1.2  will  assist  the  Program  Manager/Team  Leader  in  assembling  the  most 
appropriate  team  in  this  planning  effort.  Three  steps  suggested  to  build  the  roster: 

1.  For  both  Core  Team  and  Augmentee  roles,  identify  all  stakeholder 
organizations  necessary  to  build  a  program  plan,  then; 

2.  List  Candidate  Core  Team  Members  and  their  roles/responsibilities. 

3.  List  anticipated  Augmentees,  understanding  that  this  role  will  probably  change 
as  the  planning  becomes  more  definitive. 

A  "Stakeholder"  can  be  defined  as  a  person  representing  an  organization  who  is 
actively  involved  in  the  program  or  whose  interests  may  be  positively  or  negatively 
affected  by  the  performance  or  completion  of  the  program;  or  an  organization  or  individual 
with  direct  or  indirect  influence  on  the  requirements  of  the  end-use  application  or  system. 
Typical  stakeholder  organizations  are  the  technology  developer,  industry  partner, 
acquirer,  and  end-user. 

When  identifying  the  team  needed  to  address  the  problem  at  hand,  the  focus  needs 
to  be  kept  on  the  development  of  the  Program  Plan,  as  the  list  of  people  needed  to  build  the 
S&T  Plan  are  likely  to  be  different  than  the  team  needed  to  execute  the  program. 

Once  the  Stakeholder  organizations  are  identified,  specific  core  team  members  and 
potential  augmentees  representing  the  stakeholders  need  to  be  recruited  with  a 
commitment  to  participate  in  the  planning  effort. 

Three  keys  to  a  successful  team; 

1.  Have  the  right  members  on  the  team 

2.  Keep  the  team  small  as  possible,  and 

3.  Keep  the  team  moving.  ...‘‘You  can’t  steer  a  car  that’s  not  moving’’ 

Table  1  suggests  some  typical  members  for  both  teams.  All  team  members  should 
understand  the  roles  they  are  expected  to  play.  Core  Team  members  are  critical  to  the 
program  planning  phase  and  should  commit  to  attending  meetings,  while  others  (i.e., 
augmentees)  may  play  a  supporting  role  needing  to  attend  meetings  if  and  when  required. 


CORE  MEMBERS: 

Augmentees: 

•  Team  Leader  /  Program  Manager 

•  Financial 

•  Researcher  /  Scientists 

•  Procurement 

•  Engineers 

•  Logisticians 

•  Customers 

•  "Gray  Beards” 

•  Industry  Partners 

•  Select  Subject  Matter  Experts 

Table  1.  Possible  Typical  Team  Members 
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Project  Name: _ 

Worksheet  1.2  Member  Name: _ 

Team  Make-up  /  Roles  Role: - 

Worksheet  Date: _ 

With  the  Problem  understood,  a  team  approach  (Core  plus  Augmentees) 
must  by  defined,  i.e.,  identify  the  skills  needed  to  discuss  and  refine  the  Problem 

Team  Directory,  Roles  &  Responsibilities 

Role  /Responsibility  ORG/Svmbol  Name  /  Contact  Info 


Augmentees  (includes  SMEs,  Finance,  Contracting,  etc.) 


...the  "SE  Conversation"  continues...  include  the  right  expertise  and  Stakeholders 

15 


Approved  for  public  release. 


□  Worksheet  1.3  -  Establish  Team  Charter 


A  "Planning"  Team  Charter  should  capture  the  problem  statement,  document  roles 
and  responsibilities  of  the  team  members,  and  set  the  planning  process  duration.  When  all 
team  members  agree  to  the  details,  you  have  a  Planning  Team  Charter  with  a  better  chance 
of  success  in  program  planning  and  execution. 

Formality  of  this  charter  varies  with  the  program  at  hand.  For  basic  research  or  in- 
house  development  projects,  an  informal  charter  might  suffice.  Advanced  Technology 
Demonstration  (ATD)  and  approved  High  Visibility  Programs  (HVPs)  have  formal  charter 
requirements  as  defined  in  AFRL  instructions.  Experience  has  shown  the  time  spent  in 
addressing  member  responsibilities  will  help  define  the  specific  expectations  for  all  the 
members  on  the  team. 

Goal  /  Objectives 

The  Team  Charter  Agreement  establishes  the  "goal"  of  the  parties  to  plan  an 
executable  program.  Execution  of  the  resulting  program  is  a  totally  separate  effort 
although  most  of  the  planning  team  members  may  continue  on  as  execution  IPT  members. 

Schedule 

It’s  important  to  get  a  "team"  agreed-upon  schedule  to  establish  the  duration  of  the 
S&T  Planning  Process  along  with  the  number  or  frequency  of  the  meetings. 

Resources 

Document  any  resources  available  to  the  team  and  effort. 

Authority  /  Accountability 

Official  recognition  of  the  direction  to  conduct  the  work. 

Signatures 

Signatures  solidify  the  agreement  and  commitment  of  all  parties,  which,  if  not 
attained,  can  spell  non-support  in  regards  to  time,  resources,  and  quality  of  information 
collected  and  the  conclusions  arrived  at. 


□  DOCUMENT:  Planning  Team  Charter 

Highly  recommended,  particularly  for  large  IPTs,  to  clearly  state  the  roles, 
responsibilities,  expectations,  and  contributions  of  the  members  of  the  team, 
to  get  buy-in  and  establish  ownership  of  the  team  process. 
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Worksheet  1.3 
Establish  Team 
Charter 


Project  Name: 
Member  Name: . 

Role: . 
Worksheet  Date: 


The  Core  Team  stays  with  the  entire  process.  Other  persons  ( Augmentees ) 
are  brought  in  with  needed  expertise  where  appropriate. 

Use  Worksheets  1.1  and  1.2  as  backup  to  this  activity 

“Planning”  Team  Charter 

•  Goal/Objectives: _ 


•  Schedule 

o  Meeting  frequency  (Weekly,  monthly,  etc.) _ 

O  Duration:  Start:  (mm/dd/yyyy)  _  End:  ( mm/dd/yyyy )  _ 

•  Resources  ( Available  to  the  Team):  _ 

•  Authority  /  Accountability  of  Team  Membership:  _ 

As  a  Core  Team  Member  for  this  planning  effort,  I  understand 
the  following  is  expected  of  all  Core  Team  Members: 

o  Commit  to  participate  in  all  team  meetings 

o  Complete  any  pre-meeting  "homework"  on  schedule 

o  Participate  in  the  documentation  of  each  step  before  proceeding  to  the 
next  step 


Signatures  of  all  core  planning  team  members  commitment 


1. 

4. 

2. 

5. 

3. 

6. 

(The  above  is  just  a  suggested  outline.  Expand  with  extra  pages  if  needed) 


...a  Team  Charter  formalizes  expectations,  removes  doubt,  and  improves  the  SE  “Conversation” 
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STEP  2  -  DETERMINE  REQUIREMENTS 


Step  1  Step  2  Step  3  Step  4  Step  5 


Fonn  > 

Detennine 

Generate 

.  Evaluate  5 

Deliver 

Team 

Requirements 

Alternatives 

Alternatives 

S&TPIan 

Do: 

•  Define  Requirements 

•  DefmeTechnology 

Challenges 

•  Define  S&T 

ExitCritena 

•  Validate  wf  Customer 

•  Understand  Relevant 

•  Assess  Alternatives 

•  Fmalire  AF  Problem 

•  Genef  3te  Alternative 

•  Select  Preferred 

Document: 

G  Requirement  Set 

□  Full  Description  of 

a  Value  Analysis 

□  AF  Problem  t  Goal 

□  Team  Charter 

•  Performance 

-  Affordability 

-  ProduciMity 

■  Supportability 

•  Others 

□  Tech  Challenge  List 

□  S&T  Exit  Cntena  Set 

□  Rationale  for 

□  Preferred  Solufion(s) 

□  Action  Pl3n 

Figure  6.  STEP  2  -  DETERMINE  REQUIREMENTS 


In  Step  2  the  team  completes  the  definition  of  the  problem  space.  Leveraging  on  the 
expertise  in  the  team,  the  PM/Team  Lead  should  direct  a  discussion  that  starts  with 
identifying  the  customer  requirements,  distilling  those  requirements  into  technical 
challenges  and  finally  assembling  a  set  of  S&T  exit  criteria.  This  discussion  is  frequently 
an  iterative  process,  as  each  step  reveals  more  information  that  affects  the  knowledge 
previously  gained. 

It  is  always  better  to  have  the  customer  involved  when  identifying  requirements. 
Throughout  the  requirements  identification  process,  assumptions  may  be  made  that  will 
need  to  be  verified  with  your  customer  representative. 

For  relatively  low  TRL  S&T  programs  with  several  potential  customers  and  a  more 
generic  problem  space,  problem  owners/customers/end  users  may  be  hypothesized  by  the 
S&T  planning  team.  Communication  with  potential  customers  is  always  a  plus,  even  at  the 
basic  research  level  of  planning.  For  these  programs,  the  S&T  planning  team  may  have  to 
formulate  a  set  of  "stretch"  requirements  applicable  to  the  future  Air  Force  using  their  best 
judgment. 
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Definitions  of  each  of  the  steps  will  help  the  team  focus  their  discussion. 

Requirement  -  a  parameter,  condition,  necessity  that  can  be  measured  and  verified 
in  a  test  demonstration. 

Technical  Challenge  -  the  reason  why  the  customer’s  requirement  isn’t  available 
today.  What  is  the  technical  "hurdle”  being  addressed? 

S&T  Exit  Criteria  -  technical  accomplishments  achieved  by  a  laboratory  S&T 
program  that  are  specific,  measurable,  and  which,  when  achieved,  signify  successful 
completion  of  the  S&T  program  (i.e.,  technology  ready  for  next  development  phase, 
whether  internal  or  external  to  AFRL). 

Validate  (Validation)  -  a  confirmation  or  endorsement,  preferable  from  a 
User/Customer,  of  the  logic,  arguments,  and  specifications  drawn  in  the  team’s  postulation. 

Requirements  are  usually  stated  in  terms  of  a  threshold  (i.e.,  acceptable)  level  and 
an  objective  (desired)  level.  However,  with  some  S&T  efforts,  customer  requirements  may 
not  have  formal  thresholds  and  objectives,  but  there  are  still  limits  at  which  the  customer 
peak  and  lose  interest. 

For  example,  a  customer  requirement  might  be  to  increase  the  range  of  a 
missile  by  20%.  In  pure  acquisition  terminology,  that  range  increase  isn’t  a  formal 
requirement  at  this  time,  but  merely  what  the  customer  desires.  Through  the  use  of 
Subject  Matter  Experts  (SMEs)  or  Modeling  and  Simulation  (M&S)  tools,  the 
customer,  together  with  the  team,  could  learn  that  improvements  less  than  15% 
have  no  benefit.  Therefore,  a  threshold  would  be  a  15%  improvement,  and  the 
objective  remains  20%.  At  the  same  time,  they  might  learn  that  increases  between 
20%  and  30%  have  diminishing  payoff,  so  after  30%,  the  customer  has  no  interest. 

Continuing  this  example  to  the  next  step  of  identifying  the  technical  challenges,  the 
team  applies  the  principles  of  missile  design  to  distill  the  range  requirement  into  the 
technical  constituents  of  missile  range,  asking  "What  is  preventing  that  requirement  from 
being  achievable  today?"  A  technical  challenge  to  increased  range  might  be  to  increase  the 
pressure  capability  of  a  composite  case  without  increasing  the  weight. 

Each  technical  challenge  needs  to  be  translated  into  specific  S&T  exit  criteria.  For 
this  example,  the  S&T  exit  criteria  might  be  associated  with  the  fiber  strength,  resin 
properties,  winding  method,  or  other  aspects  of  the  component  design.  The  scientist  and 
engineer  cannot  assume  the  customer  will  understand  this  translation  of  the  problem  and 
should  keep  the  communication  open  by  providing  the  translation  between  the  user’s 
language  and  the  technical  language. 

At  this  point  in  the  team  process,  it  is  recommended  that  the  proposed  requirement 
set  be  validated  by  the  customer  representative,  if  they  are  not  already  active  on  the  team. 
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While  engaging  the  customer  in  the  development  of  the  requirements  during  the 
iterative  discussions  is  very  beneficial,  that  involvement  can  lead  to  another  negative 
phenomenon:  Requirements  Creep!  This  occurs  as  the  team  or  the  customer  become 
excited  with  the  possibilities  of  the  new  capability,  and  seeks  to  push  the  requirements 
beyond  what  was  originally  established.  Once  a  requirement  has  been  developed  and  the 
measurable  parameters  are  identified,  the  team  needs  to  exercise  a  requirements 
management  process  to  guard  against  requirements  creep. 

Note:  in  many  cases,  requirement  definition  requires  more  than  one  iteration  of  the 
streamlined  process.  In  a  hypothetical  example  portrayed  in  Table  2,  one  can  see  a 
comparison  after  a  first  and  second  application  of  the  Streamlined  Planning  Process.  In  this 
example,  the  first  quick  pass  was  not  accomplished  at  a  sufficient  level  of  detail,  resulting  in 
a  lack  of  definition  of  desired  Warfighter  capabilities  and  quantitative  metrics.  A  second, 
more  in-depth,  application  of  the  process  results  in  a  much  clearer  quantitative  definition 
of  customer  expectations. 


First  ADDlication 

After  Second  Annlication 

•  Attribute-level  metrics  based  on 

•  Attribute-level  metrics  based  on 

expected  characteristics  (i.e.,  output 

Warfighter  desired  capabilities  (i.e., 

power,  efficiency,  etc.) 

range,  time-on-station,  etc.) 

•  Product-level  metrics  based  on 

•  Product-level  metrics  based  on 

supporting  expected  characteristics 

providing  the  attribute-level 
capabilities 

•  Examples: 

•  Examples: 

o  Efficiency:  10%  (what  we  can 

o  Efficiency:  20% 

do  currently  today) 

o  20  nm  typical  target  slant 

o  No  effective  range  metric 

range 

o  No  magazine  depth  metric 

o  100  target/sortie  magazine 

o  No  weight  or  volume  metric 

depth 

o  No  probability  of  effect  metric 

o  Less  than  15,000  lbs  total 
weight 

o  Less  than  5%  probability  of 
second  shot  required  to 
achieve  effect 

Table  2:  Hypothetical  S&T  Program  Requirements  after  first  and  second  application 
of  Tailored  Systems  Engineering  (improved  metrics  depicted  in  blue  text). 
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Each  requirement  should  be  carefully  reviewed  to  determine  it  can  eventually  be 
satisfied  by  a  technical  solution.  Depending  on  the  complexity  of  the  requirement,  this  may 
or  may  not  be  accomplished  during  the  S&T  program.  When  RX  technology  developments 
are  integrated  into  components,  such  as  a  turbine  engine  component,  testing  of  the 
component  is  frequently  part  of  a  follow-on  S&T  Program  by  another  AFRL  directorate  or  a 
follow-on  acquirer/developer  before  delivery  to  the  end  user. 

There  is  a  possibility  that  once  the  team  begins  developing  this  requirement  set, 
they  realize  the  team  is  missing  a  needed  skill  set  or  role.  That  is  the  iterative  nature 
inherent  in  this  problem-solving  process,  and  should  not  be  rejected. 

A  SE  facilitator  can  help  lead  a  requirements  generation  review  of  the  problem 
statement,  exploring  what’s  known,  not  known,  and  identify  the  technical  challenges 
resulting  from  the  requirements.  During  Step  2,  the  SE  facilitator  can  help  the  team 
members  update  and  prioritize  (high,  medium,  low)  requirements  and  check  the  list  for 
completeness.  As  the  requirements  are  translated  into  technical  challenges  and  S&T  exit 
criteria,  the  SE  facilitator  can  assist  the  team  members  with  appropriate  tools  to 
sufficiently  describe  the  S&T  exit  criteria  for  measurability  and  testability. 

The  worksheet  activities  of  Step  2  are  designed  to  help  the  team  get  the 
requirements,  technical  challenges,  and  the  derived  S&T  exit  criteria  down  on  paper.  The 
better  these  are  identified  and  validated,  the  greater  the  chance  of  program  success. 

The  final  documents  produced  during  Step  2  are  the  refined  customer  requirements, 
technical  challenges,  and  prioritized/weighted  S&T  Exit  Criteria,  which  completes  the 
definition  of  the  problem  space. 


'Warfighters  are  known  to  have  Alligator  Eyes  and  Chicken  Wallets." 

....Attributed  to  a  Command  Staffer 
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Step  2 


Determine 

Requirements 


•  Define  Requirements 

•  Define  Technology 

Challenges 

•  Define  S&T 

Exit  Criteria 

•  Validate  w/  Customer 


□  Requirement  Set 

-  Performance 

-  Affordability 

-  Producibility 

-  Supportability 

-  Others 

□  Tech  Challenge  List 

□  S&T  Exit  Criteria  Set 


Step  2  -  Determine  Requirements 

(See  Streamlined  SE  Process  Workbook  for  suggested  Worksheets) 


□  Worksheet  2.1  -  List  of  Customer  Requirements 

□  Worksheet  2.2  -  Technical  Challenges 

□  Worksheet  2.3  -  List  of  S&T  Exit  Criteria 

□  Worksheet  2.4  -  Group  S&T  Exit  Criteria  by  Category 

□  Worksheet  2.5  -  S&T  Exit  Criteria  Complete  Description 


Homework  - 

In  advance  of  a  team  requirements  meeting,  take  a 
moment  and  fill  out  the  first  three  Step  2  Worksheets  with 
what  you  think  you  know  about  the  customer’s 
requirements,  the  technical  challenges  to  overcome,  and 
the  threshold  and  objective  values  to  measure  success. 


"Gentlemen,  we  have  run  out  of  money.  Now  we  have  to  think"  (Winston  Churchill] 
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□  Worksheet  2.1  -  List  of  Customer  Requirements 

Worksheet  2.1  is  available  to  guide  the  PM  or  team  through  a  deductive,  iterative 
process  of  developing  and  qualifying  customer  requirements.  As  the  understanding  of  the 
problem  space  and  customers’  desires  evolve,  the  ability  to  best  identify  and  quantify  the 
requirements  for  the  project  improves.  During  the  process,  the  team  may  need  to 
periodically  return  and  modify  the  requirements  list. 

Requirement:  The  set  of  all  (customer)  expectations  for  procured  products  and/or 
services  where  each  expectation  is  expressible  in  terms  of  a  formalized  desirement. 

Good  customer  requirements  are  well-crafted  characteristics  that  describe  a  desired 
capability  and  generally  fall  into  common  areas  such  as  performance,  cost,  scheduled 
availability,  affordability,  producibility,  reliability,  and  supportability. 

Requirement  Name 

Use  Worksheet  2.1  to  enter  a  descriptive  name  for  each  Requirement. 

Requirement  Description 

Provide  as  much  quantified  information  to  describe  the  requirement  as  available. 

Threshold 

After  listing  the  requirement(s),  determine  the  incremental  acceptable  Threshold 
value(s)  that  must  be  achieved  for  the  technology  to  advance  to  the  next  stage  of 
development  or  transition  (usually  stated  as  a  minimum  or  maximum  value). 

Objective 

Determine  the  ultimate,  end-state  capability  as  the  Objective  value.  This  is  where  a 
customer  representative(s)  is  so  important  to  validate  the  needs,  conditions,  and  desires. 
The  stretch-goal  value  that  is  desirable,  but  not  essential  (often  viewed  as  the  requirements 
trade  space  if  parameters  such  as  cost  or  schedule  are  more  important  to  the  customer). 


□  DOCUMENT:  Requirements  Set 

Activity  generates  a  set  of  desired  capability  requirements. 


" Everything  looks  like  a  nail  if  you’re  a  hammer"  -  Anonymous 
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Step  2:  Requirements.  A  solid  understanding  of  the  requirements,  technical  challenges,  and 
how  they’re  stated  as  S&T  Exit  Criteria  are  critical  steps  in  Problem  Space  documentation. 


Worksheet  2.1  Project  Name: _ 

.  r  _  Member  Name: _ 

List  of  Customer  „ . 

Role: _ 

Requirements  Worksheet  Date: _ 

The  Core  Team  refines  the  understanding  of  the  Customer’s  Problem 
into  stated  requirements  and  validates  them  with  the  Customer 


Requirement 

Name 

Requirement  Description 

(Be  as  specific  as  possible] 

Threshold* 

Objective* 

*Desirable  -  vary  with  maturity  of  task 


Threshold:  The  measurable  value  that  must  be  achieved  for  the  technology  to  advance  to  the  next 
stage  of  development  or  transition  ( usually  given  as  a  minimum  or  maximum  value]. 

Objective:  The  stretch-goal  value  that  is  desirable,  but  not  essential  (often  viewed  as  the 
requirements  trade  space  if  parameters  such  as  cost  or  schedule  are  more  important  to  the  customer). 

□  DOCUMENT:  Validate  this  worksheet  with  your  customer  representatives /stakeholders 
to  ensure  agreement  with  the  goals. 


...the  “Conversation”  continues...  What  capability  does  my  Customer  (Want)  Need?  ...and  Why? 
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□  Worksheet  2.2  -  Technical  Challengers) 


With  the  customer  requirements  defined,  the  PM  or  team  needs  to  identify  the 
technical  challenges  that  are  appropriate  for  the  level  of  S&T  effort  being  planned. 
Technical  Challenges  are  the  technical  hurdles  that  must  be  overcome  to  provide  a  new 
and/or  improved  capability. 

The  amount  of  rigor  or  effort  involved  in  this  step  varies  depending  on  the 
complexity  of  the  problem  and  the  expectations  of  the  Stakeholders.  The  effort  can  range 
from  a  guided  discussion  by  the  core  team  members,  to  a  formal  Goals,  Objectives, 
Technical  Challenges,  and  Approach  (GOTChA)  analysis  or  other  formal  technical  planning 
methodology. 

What  might  be  Technical  Challengers)? 

A  well  worded  Technical  Challenge  answers  the  question,  "What  is  preventing  us 
from  achieving  the  stated  objective  today?"  For  example,  but  not  inclusive: 

•  Physical  constraints:  Min/Max  size,  Min/Max  weight,  Min/Max  transportability 

•  Power  constraints  -  Min/Max  power  required,  Min/Max  conversion  efficiency 

•  Properties:  Min/Max  strength,  Min/Max  thickness,  Min/Max  flexibility 

•  Coatings  -  Min/Max  thickness,  Min/Max  wear,  Min/Max  conduction,  corrosion 

Requirement  Name 

Carry  forward  the  Requirement  Name  from  Worksheet  2.1  into  Worksheet  2.2. 

Technical  Challenge  Issues 

Use  worksheet  2.2  to  describe  the  Technical  Challenge  Issues  in  as  much  detail  as 
possible.  The  issues  stated  here  will  help  derive  accurate  S&T  Exit  Criteria  in  the  next 
worksheet  activity. 


□  DOCUMENT:  Technical  Challenge(s) 

In  any  S&T  effort,  there  should  be  some  technological  hurdle  to  overcome;  otherwise, 
the  technology  would  be  developed  in  the  commercial  world.  The  discussions  and 
documentation  of  the  technical  challenges  should  help  steer  the  team  to  better 
understand  the  requirement  and  to  better  state  the  S&T  Exit  Criteria. 


For  additional  references  on  identifying  Technical  Challenges,  see:  RX  Systems  Engineering 

COP  https: //livelink.ebs.afrl.af.mil/livelink/llisapi.dll?func=ll&obiId=  1708 1764&obiAction=browse&sort=name&viewTvpe=l 
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Worksheet  2.2 
Technical  Challenges 


Project  Name: . 
Member  Name: . 

Role: . 
Worksheet  Date: 


The  Team  now  has  to  turn  the  Customer  Requirements  into  actionable 
S&T  descriptions  (S&T  Exit  Criteria ).  This  step  helps  the  team  decompose 
the  Customer  Requirements  into  Technical  Issues  ( Challenges ). 


Requirement  Name 

Technical  Challenge  Issues 

....the  SE  Conversation  continues  by  identifying  "Where's  the  S&T  in  this  challenge?". ..and  Why? 
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□  Worksheet  2.3  -  List  S&T  Exit  Criteria 


Worksheet  2.3  is  designed  to  define  the  S&T  Exit  Criteria  associated  with  each 
stated  requirement  and  the  derived  technical  challenges  in  terms  of  physical,  measurable 
attributes;  and  stated  as  a  threshold  (acceptable)  and  objective  (goal)  level.  As  part  of  the 
S&T  Exit  Criteria,  the  desired  means  of  demonstrating/validating  appropriate  TRL  and 
MRL  levels  (See  Appendix  H)  have  been  achieved  should  be  described. 

As  with  Technical  Challenges,  there  is  not  necessarily  a  one-to-one  relationship 
between  the  requirements  defined  in  Step  2.1  and  the  S&T  exit  criteria  defined  here.  Some 
requirements  may  need  several  S&T  exit  criteria  defined,  whereas  other  S&T  exit  criteria 
may  meet  several  user  requirements  for  a  given  problem.  However,  it  is  valuable  to  verify 
that  all  requirements  containing  technical  challenges  map  to  at  least  one  S&T  exit  criterion, 
and  all  S&T  exit  criteria  map  to  at  least  one  requirement. 

S&T  Exit  Criteria,  which  include  S&T  Key  Performance  Parameters  (KPPs),  are 
levels  of  measurable  performance  (thresholds)  that  must  be  achieved  for  program  success. 

•  Research  program,  where  successful  completion  allows  entry  into  the  next 
program  development  phase. 

•  Development  product  has  achieved  an  expected  level  of  functionality,  where 
successful  achievement  allows  entry  into  the  next  development  phase. 

Criteria  Name 

Name  may  very  well  be  the  same  as  the  requirement  or  it  may  be  more  specific. 

Description 

What  are  the  key  performance  criteria  for  to  determine  success? 

Threshold  /  Objective 

Threshold  /  Objective  should  match  the  same  values  as  previously  documented  on 
Worksheet  2.1,  Customer  Requirements  List. 

RQT  Name 

Correlate  the  requirement  name  to  the  criterion.  In  some  cases,  there  may  be 
multiple  criteria  for  a  single  requirement. 


□  DOCUMENT:  S&T  Exit  Criteria 

S&T  Exit  Criteria  should  be  validated  against  all  requirements  and  tech  challenges. 
Validation  can  range  from  simple  confirmation  on  an  early  program,  to  ATDs  where 
requirements  and  criteria  are  fully  documented  and  signed  by  all  participants. 
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Worksheet  2.3 
List  of 

S&T  Exit  Criteria 


Project  Name: 
Member  Name: . 

Role: . 

Worksheet  Date: 


Customer  Requirements  are  usually  at  a  fairly  high  level.  The  team  now  has  to  define  and 
document  what  a  final  product,  technology,  or  system  must  do,  including  the  parameters 
that  define  successful  completion.  Worksheets  2.1  and  2.2  should  lead  into  this  step. 


Criteria  Name 

Description 

Threshold* 

Objective* 

RQT  Name 

*Desirable  -  vary  with  maturity  of  task 

Threshold:  The  measurable  value  that  must  be  achieved  for  the  technology  to  advance  to  the  next 
stage  of  development  or  transition  (usually  given  as  a  minimum  or  maximum  value). 

Objective:  The  stretch-goal  value  that  is  desirable,  but  not  essential  (often  viewed  as  the 
requirements  trade  space  if  parameters  such  as  cost  or  schedule  are  more  important  to  the  customer). 


...the  “Conversation"  continues...  What  S&T  Exit  Criteria  demonstrates  the  Requirement(s)  have  been  met?"  - 

....and  Why? 
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□  Worksheet  2.4  -  Group  S&T  Exit  Criteria  by  Category 


With  the  requirements  captured,  the  technical  challenges  understood,  and  the  S&T 
Exit  Criteria  stated,  the  S&T  Exit  Criteria  should  be  grouped  by  like  type  (categories),  such 
as  cost,  schedule,  and  performance.  Grouping  of  the  Exit  Criteria  may  be  difficult  but  the 
results  are  usually  worth  the  effort. 

Here  is  a  suggested  list  of  groupings  to  add  to  or  delete: 

•  Cost:  Acquisition,  Deployment,  Operating  Cost  at  Point  of  Use,  Development  Cost 

•  Schedule:  FY  (Fiscal  Year)  required  for  transition;  FY  required  for  delivery  to 
another  TD  for  subsequent  development 

•  Performance:  Power,  Reliability,  Weight,  Footprint,  Set-up  Time,  Diagnostics, 
Repair,  Surge  Capacity-Spike/-Continuous,  Improved  System  Efficiency 

•  Logistics:  Air/Ship/Truck/Rail  Transportability,  Service  Life,  Storage  Costs, 
Preventative  Maintenance  Inspections,  Scalability,  Disposal 

•  Human  Factors:  Skill  Level  Required  for  Set-Up/Use,  Man-Hours 

•  Environmental:  Emissions,  certifications,  disposal 

•  Strategic:  Significant  or  Unique  Operational  Need  or  Unique  Technical  Competency 

•  Political:  Outside  influences.  (For  example,  a  host  country  wanting  deployed  base 
structures  to  appear  temporary,  or  not  permanent,  i.e.,  a  Valley  Forge  "Tent-City" 
appearance,  so  as  not  to  cause  local  populations  to  feel  the  occupation  is  long-term) 
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Worksheet  2.4 
Group  S&T 
Exit  Criteria 
by  Category 

S&T  Exit  Criteria  negotiated  by  the  team  can  usually  be  grouped  into  common  topics, 
identified  such  as  P-1,  P-2,  etc.  for  Performance;  C-l,  C-2,  etc.  for  Cost. 

There  is  no  limit  to  the  number  of  S&T  Exit  Criterion,  but  this  is  the  logical  place 
to  consider  consolidation,  combining  the  criteria,  if  appropriate. 


ID 

S&T  Exit 

Criteria 

Description 

Group: 

Group: 

Project  Name: 
Member  Name: . 

Role: . 

Worksheet  Date: 


Possible  Category  Groupings  (add  more  type  categories  depending  on  your  technology) 

C=  Cost  (development,  acquisition,  deploy,  point  of  use  operation)  HF  =  Human  Factors  (Skill  Level) 

S  =  Schedule  E  =  Environment  (EPA,  certifications,  etc.) 

P  =  Performance  (reliability,  weight,  footprint,  set-up  time)  ST  =  Strategic  (outside  influences,  be  they  political  or  otherwise) 

L  =  Logistics  (transport,  service  life,  storage,  scalability,  disposal  POL  -  Political  (constrictions) 


...the  "Conversation"  continues...  What  groups  form  the  S&T  Exit  Criterion?"  -....and  Why? 
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□  Worksheet  2.5  -  Group  S&T  Exit  Criteria  Complete  Description 

The  last  Step  2  activity  is  to  compile  the  information  into  a  Scorecard  or  complete 
description  of  the  S&T  Exit  Criteria. 


ID 


Name 


Priority 

/Wt* 


Units 


Threshold 

Value 


Threshold 

Rationale 


Objective  Objective 
Value  Rationale 


How 

Measured 


ID 


Method  to  track  identification,  i.e.,  C-l,  C-2,  C-3,  P-1,  P-2,  etc.  as 
defined  in  Worksheet  2.4 


Name 

Priority  /  Wt 


Units 

Threshold  Value 

Threshold  Rationale 
Objective  Value 


Recognizable  title  identifying  the  thrust  of  the  concept 

Either/Or,  Priority,  sorts  the  list  from  the  most  important  to  the 
least  important,  possibly  in  a  High-Med-Low  ranking;  Weight  is  a 
relative  value  to  the  importance  of  the  contribution,  for  example, 
score  a  weight  value  of  1.0  if  the  criteria  is  critical  to  the 
capability,  0.7  if  the  criteria  is  important,  and  0.5  if  not  as 
important. 

The  component  or  scale  of  measurement  being  used,  for 
example,  miles  per  gallon,  distance,  weight,  time,  decibels,  etc. 

An  incremental  value  that  demonstrates  a  targeted  difference 
never  attained  before,  usually  a  stepped  improvement. 

The  reason  the  Threshold  Value  was  chosen  as  an  incremental 
step  given  the  maturity  of  the  technology. 

The  ultimate/target  value  providing  real  capability  once 
attained. 


Objective  Rationale  The  reason  the  Objective  value  was  chosen  as  an  incremental 

step  given  the  maturity  of  the  technology. 

How  Measured  How  will  the  alternatives  selected  be  measured  demonstrating 

the  requirements  have  been  met? 
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Worksheet  2.5 
S&T  Exit  Criteria 
Complete  Description 


Project  Name: _ 

Member  Name:  _ 

Role: _ 

Worksheet  Date: 


The  Team  defines  the  detailed  quantitative  attributes  of  the  S&T  Exit  Criteria. 


ID 

Name 

Priority 

/Wt* 

Units 

Threshold 

Value 

Threshold 

Rationale 

Objective 

Value 

Objective 

Rationale 

How 

Measured 

Group  -- 

Group  -- 

*Can  be  used  to  compare  relative  importance  of  S&T  Exit  Criteria.  These  are  suggestive,  use  whatever  scale  you 
determine  relevant/needed” 

Priority  qualitative  ranking:  High,  Medium,  Low,  ....Or 

Weight  scores  contribution  importance:  -1=  critical  contribution  to  capability,  0.7  important  contribution,  and 
0.5  if  not  too  important 

Threshold:  The  measurable  value  that  must  be  achieved  for  the  technology  to  advance  to  the  next  stage  of 
development  or  transition  (usually  given  as  a  minimum  or  maximum  value). 

Objective:  The  stretch-goal  value  that  is  desirable,  but  not  essential  (often  viewed  as  the  requirements  trade 
space  if  parameters  such  as  cost  or  schedule  are  more  important  to  the  customer). 


...the  “Conversation”  continues...  When  is  ‘more  or  less'  better  and  how  do  you  measure  it?” -....and  Why? 
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STEP  3  -  GENERATE  ALTERNATIVES 


Step  1  Step  2  Step  3  Step  4  Step  5 


Do: 

•  Understand  Relevant 
State-of-the-Art 
Technologies 

•  Identify  All 

•  Define  Technology 

•  Validate  wif  Customer 

•  Generate  Alternative 
Solutions 

•  Select  Preferred 

Document: 

□  Problem  Definition 

□  Requirement  Set 

□  Full  Descnption  of 

a  Value  Analysis 

□  AF  Problem  t  Goal 

uj  icoiii  nuatci 

each  Aitematrve, 
including 

□  Preferred  Solution(s) 

□  Team  Charter 

□  Tech  Challenge  List 

□  S&T  Exit  Cntena  Set 

■  TRL  Levels 

•  MRl  Levels 

•  Schedule  /  Key 

Milestones 
-  Cost  Estimate 

□  Rationale  for 

□  Action  Pl3n 

Figure  7.  STEP  3  -  GENERATE  ALTERNATIVES 


In  Step  3,  the  planning  team,  possibly  assisted  by  a  SE  Facilitator,  will  generate  a  list 
of  alternatives  (documented  on  Worksheet  3.1)  that  might  solve  the  problem  as  defined  in 
Steps  1  and  2.  The  Step  3  activity  might  be  accomplished  in  a  single  meeting  for  a  simple 
problem;  or  the  team  might  require  several  meetings  depending  on  the  complexity.  NOTE, 
a  pitfall  of  conducting  this  activity:  the  results  of  alternative  generation  frequently  reflect 
the  area  of  expertise  of  the  people  on  the  team;  for  example,  an  all-RX  team  is  likely  to 
generate  an  all-material  solution.  A  multi-directorate  approach  is  very  desirable  to  solve 
critical  Warfighter  problems.  Alternatives  need  to  fit  within  the  problem  space  defined  by 
the  S&T  exit  criteria,  but  remain  unconstrained  in  creativity  and  approach. 


Allow  creativity.  Avoid  the  ‘quick  leap' 
to  familiar  technology  solutions 


The  previous  steps  provided  the  team  some  insight  into  why  the  current  process  or 
way  of  operating  was  no  longer  acceptable  to  the  customer.  In  order  to  offer  an 
improvement  to  the  customer,  the  team  needs  to  be  familiar  with  the  current  process  or 
state  of  the  art.  Different  problems  will  be  dealing  with  different  periods  of  appropriate 
responses;  ranging  between  near-,  mid-,  and  far-term. 
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First,  let’s  define  the  term  “performance  baseline."  The  baseline  is  viewed  as  the 
way  the  user  is  currently  addressing  the  problem,  or  the  current  technology,  doctrine, 
operations  or  training  approach  used  to  meet  the  user’s  need. 

The  planning  team  must  be  aware  of  recent  developments  in  relevant  technology 
areas  across  the  scientific  community,  including  AFRL,  DOD,  other  government  agencies 
such  as  NASA  and  DOE,  academia,  and  industry.  For  late  6.2  or  6.3  technology  efforts, 
knowledge  of  the  "State-of-the-Art"  (SOTA)  will  help  the  planning  team  identify  near-term 
technologies  that  have  the  potential  of  transitioning  into  an  Air  Force  system  within  a 
reasonable  timeframe.  Current  state,  or  SOTA  knowledge,  is  constantly  gained  via 
continued  education,  literature  searches,  guest  speakers,  technology  briefings,  conferences, 
symposiums,  and  experimentation. 

Next,  the  team  must  determine  if  AFRL/RX  is  in  a  leadership  position  in  the 
particular  technical  area  under  consideration.  From  both  an  in-house  research  and 
program  management  viewpoint,  leadership  in  the  technology  area  with  knowledge  of  the 
specific  customer  requirements  and  S&T  exit  criteria  (generated  in  Step  2),  is  crucial  to  the 
success  of  the  team  in  solving  the  problem. 

If  it  is  determined  RX  possesses  the  technical  competency  to  invest  in  the  technical 
area(s)  under  consideration,  then  the  planning  team  should  be  ready  to  continue  to 
conduct  the  "Generate  Alternative  Solutions"  activity  by  listing  on  them  Worksheet  3.1. 

Discussing  the  SOTA  and  Near-Term  Technologies  with  the  team  at  the  beginning  of 
the  Generate  Alternatives  step  can  be  a  good  way  to  bring  the  team  up  to  a  same  level  of 
knowledge  about  the  Problem-Solution  space  and  will  help  the  team  to  generate  better 
solutions  to  the  problem(s)  at  hand. 


Generate  Alternatives 
in  the  S&T  Environment 


Brainstorming  is  traditionally  viewed  as  part  of  a  Teambuilding  process  that 
challenges  the  group  participants  to  "step  outside  of  their  comfort  zones"  and  strengthen 
team  problem  solving  skills.  The  idea  is  to  suspend  critical  thinking  and  generate  a  number 
of  ideas  that  represent  potential  solutions  that  might  provide  breakthrough  program 
product  or  process  results. 
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The  importance  of  the  alternatives  generation  activity  is  to  ensure  awareness  of  all 
relevant  research  and  not  make  the  'quick  leap'  decision  to  a  given  technology  solution 
without  at  least  considering  viable  alternative  approaches. 

Often,  the  alternative  solution  set  consists  of  several  technologies  with  different 
levels  of  maturity  (i.e.,  technical  risk),  but  offer  different  levels  of  payoff  to  the  customer,  as 
suggested  in  the  Figure  7.  Alternative  A  might  offer  great  potential  payoff,  but  is  early  in 
technology  maturity  and  has  potentially  greater  risk,  while  Alternative  C  is  mature,  ready 
to  transition,  but  has  limited  payoff  potential. 

Alternative  A 

A 


£= 

cu 

+-> 

o 

Q_ 


Alternative  B 

A 


Alternative  C 

A 


Technology  Maturity  (TRL) 

Figure  8.  Comparison  of  Alternative  Solutions 


Although  all  three  solutions  might  meet  the  S&T  Exit  Criteria,  considerations  such  as 
S&T  development  cost  and  timeframe,  along  with  transition  cost  and  timeframe  may  drive 
the  planning  team  to  a  particular  solution  in  the  evaluation  step  of  this  process. 


It  is  common  for  AFRL  Scientists  and  Engineers  to  concentrate  on  the  materiel 
solutions  to  solve  a  problem.  However,  sometimes  customers  come  to  AFRL 
looking  for  answers  to  solve  their  problems,  when  a  change  to  their  current 
operations,  training  or  logistics;  a  non-material  solution  is  still  a  viable  option. 
The  team  should  be  comfortable  in  going  back  to  the  customer  and  evaluating 
non-materiel  solutions  before  proceeding  with  a  technology  development. 
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Step  3 


Generate 

Alternatives 


•  Understand  Relevant 
State-of-the-Art 

Technologies 

•  Generate  Alternative 

Solutions 


□  Full  Description  of 
each  Alternative, 
including: 

-  TRL  Levels 

-  MRL  Levels 

-  Schedule  /  Key 

Milestones 

-  Cost  Estimate 


Step  3  -  GENERATE  ALTERNATIVES 

(See  Streamlined  SE  Process  Workbook  for  suggested  Worksheets) 


□  Worksheet  3.1  -  List  Alternatives 

□  Worksheet  3.2  -  Alternatives  Complete  Description 


Homework: 

Before  the  Step  3  "Alternatives  Generation"  Team  Meeting, 
each  team  member  should  attempt  to  complete  worksheets 
3.1  and  3.2  from  their  own  expertise. 


"Logic  will  get  you  from  A  to  B.  Imagination  will  take  you  everywhere"  -  Albert  Einstein 
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□  Worksheet  3.1  -  List  Alternatives 


At  this  point  in  the  process,  the  team  of  Scientists  and  Engineers  can  be  released  to 
unleash  their  creative  genius  and  propose  ideas  to  solve  the  technical  problem.  As 
previously  mentioned,  filling  out  Worksheet  3.1  is  likely  to  be  an  iterative  process,  as  a 
detailed  description  of  the  potential  solutions  is  likely  to  require  some  elements  of 
discovery.  A  process  facilitator  may  be  a  good  resource  to  invite. 

Worksheet  3.1,  Team  Members  should  suggest  technology  concepts/alternatives 
which  could  apply  to  the  problem  as  defined  by  the  conclusion  of  Step  2.  AFRL  Scientists 
and  Engineers  have  always  been  good  at  creative  technical  solutions;  this  step  in  the 
process  is  to  help  focus  that  creativity  to  the  problem  at  hand. 

Alternative  Name 

Starting  with  the  S&T  Exit  Criteria,  each  Team  Member  should  identify  technology 
solutions  which  have  the  potential  to  solve  the  technical  challenges  represented  by  the  Exit 
Criteria.  While  evaluating  Alternatives  is  not  to  occur  at  this  point  in  the  process,  the 
essential  elements  of  each  alternative  should  include  an  understanding  of  how  each 
alternative  addresses  the  S&T  Exit  Criteria,  the  anticipated  performance  of  the  alternative 
relative  to  the  Exit  Criteria,  and  the  sense  of  confidence  or  uncertainty  around  that 
anticipated  prediction. 

Questions  to  spur  the  team’s  thinking: 

•  What  option/alternatives  might  solve  the  stated  problem? 

•  Explore  "What  If?"  and  "Why  Not?"  (Modeling,  Experimentation,  Leverage) 

•  Consider  Near-Term/Low  Payoff  solutions  as  well  as  Far-Term/High  Payoff 
ideas. 

•  Do  not  evaluate  or  discount  any  idea  at  this  point 

•  Is  a  material  solution  the  only  option?  Material  solutions,  the  physical  materials 
and  tools  necessary  to  execute  any  work,  enterprise,  etc.;  specif.,  weapons, 
equipment,  supplies,  etc.  of  the  armed  forces.  Non-Material  solutions  include 
changes  to  current  Tactics,  Training,  and  Mission  Operations  to  address 
deficiencies  have  been  considered  by  the  customer  (MAJCOMs). 


□  Document:  List  Alternative  Solutions 

List  of  Alternatives  owned  by  the  team  (Worksheet  3.1 ) 
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Step  3:  Generate  Alternatives.  List  all  the  possible  ways  to  solve  a  problem. 


Worksheet  3.1 
List  Alternatives 


Project  Name: 
Member  Name: . 

Role: . 
Worksheet  Date: 


The  Team  identifies  possible  solutions  (called  Alternatives)  to  satisfy 
each  of  the  S&T  Exit  Criteria.  The  Alternatives  may  be  existing  technologies, 
but  the  team  should  also  explore  novel  approaches  that  may  offer 
enhanced  payoff...  even  if  it  seems  a  higher  risk. 


Alternative  Name 

Description 

...the  “Conversation”  continues. ..What  Alternatives  might  satisfy  the  S&T  Exit  Criteria?  ....and  Why? 
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□  Worksheet  3.2  -  Alternatives  Complete  Descriptions 

With  the  collection  of  alternatives  gathered,  it  now  falls  to  the  team’s  responsibility 
to  best  define  and  describe  each  alternative  in  as  much  detail  as  possible  for  informed 
decision  making  in  Step  4.  Worksheet  3.2  is  designed  to  provide  that  definition,  one  per 
each  alternative.  Sketches  of  each  alternative  can  also  help  the  team  visualize  the  nature  of 
the  proposed  technical  solution. 


Alternative  Name:  Enter  the  alternative  name  here 

Description:  Describe  the  alternative,  concept,  opportunity  in  as  much  detail  as  possible 

Any  Considerations:  Highlight  any  crucial  fact  of  knowledge  or  limiting  conditions 

Estimate  Cost  and  Schedule:  Any  ballpark  cost  figures  and  estimate  of  schedule 

Estimate  Initial  TRL:  Where  is  this  technology  currently? 

Estimate  Final  TRL:  Where  could  it  be  if  developed  further? 

Estimate  Initial  MRL:  What  is  the  current  manufacturing  ability  for  this  technology? 
Estimate  Final  MRL:  What  could  the  final  manufacturing  ability  be  for  this  technology? 

Estimate  Payoff  to  Maturity  score:  Based  on  Maturity,  what’s  the  impact  of  this 
technology? 

Picture  or  graphic  representation:  helpful  if  available 


NOTE:  As  with  any  of  the  offered  worksheets,  the  worksheets  can  be  customized, 
expanded  or  deleted  as  needed  depending  on  the  nature  of  the  information  required.  A 
Quad  Chart  like  presentation  may  be  desirable. 


"If  we  knew  what  is  was  we  were  doing,  it  would  not  be  called  research,  would  it?" 

-Albert  Einstein 
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Worksheet  3.2 
Alternatives 
Complete 
Descriptions 

(One  Worksheet  per  Alternative) 


Project  Name: 
Member  Name: . 

Role: . 

Worksheet  Date: 


Describe  the  “ attributes ”  of  each  Alternative  in  as  great  a  detail  as  possible  as  they  pertain  to 
each  S&T  Exit  Criterion.  Some  form  of  Description/Quad  Chart  presentation  (attached  to 
Worksheet  3.2),  could  make  value  assessment  and  decision  briefing  easier  as  you  go. 


Alternative  Name: 


Description:. 


Any  Considerations: _ 

Estimate  Cost  and  Schedule: _ 

Estimate  Initial  TRL: _  Final  TRL: _ 

Estimate  Initial  MRL: _  Final  MRL: _ 

Estimate  Payoff  to  Maturity  score:  ( Score  the  Alternative  on  the  table  below) 


Mature 

i . 1  Low  Payoff 

Alternative 

Picture  or 

Near  Term 

Graphical 

New  Capability 

Representation 

Immature 

/  Illustration 

Game-Changer 

if  available 

_ 

Payoff 
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STEP  4  -  EVALUATE  ALTERNATIVES 


Step  1 

Step  2 

Step  3 

Step  4 

Step  5 

Form 

Determine 

Generate  _ ^ 

Evaluate 

Deliver 

Team 

Requirements 

Alternatives  | 

Alternatives 

_ 

S&T  Plan 

Do: 

•  Define  Requirements 

•  Understand  Relevant 

•  Assess  Alternatives 
•  Score  S&T 

Exit  Cntena 
-  Evaluate  Risk 

•  Select  Prefenred 

Alternative(s) 

•  Prepare  for  Intended 

Document: 

□  Requirement  Set 

□  Tech  Challenge  List 

□  S&T  Ex<  Cnten3  Set 

□  Full  Description  of 

□  Value  Analysis 

-  Alternative  Scores 

-  Top  Risks  & 
Mitigation  Plans 

□  Rationale  for 

Preferred 

Altematrve(s) 

□  OO 

Figure  9:  STEP  4  -  EVALUATE  ALTERNATIVES 


In  Step  4,  the  team  compares  and  evaluates  the  potential  of  each  alternative  solution 
against  the  stated  S&T  Exit  Criteria  from  the  Problem  Space.  This  comparison  of 
alternatives  against  the  S&T  Exit  Criteria  assumes  the  team  knows  early  in  the  process 
whether  the  solutions  will  or  will  not  solve  the  problem  to  the  satisfaction  of  the  customer. 
After  that,  the  various  solutions  can  be  scored  in  terms  of  which  one  offers  the  best  value  to 
the  customer,  with  "value"  being  determined  by  the  team. 

In  order  to  evaluate  the  alternative,  the  team  needs  to  go  through  a  process  of 
discovery  for  each  alternative.  In  advance  of  the  team  meeting,  the  Team  Leader  should 
assign  each  proposed  alternative  solution  to  one  or  more  of  the  team  members,  based  on 
their  level  of  knowledge  of  the  topic.  The  T earn  Member(s),  as  the  focal  point  for  that  topic, 
needs  to  be  able  to  evaluate  the  idea  against  the  Problem  specific  S&T  Exit  Criteria  as 
defined  in  Step  2.  This  requires  more  than  just  a  knowledge  of  the  technology;  it  also 
requires  an  understanding  of  the  unique  interactions  between  the  technology  and  the 
intended  application. 

Recent  experience  with  this  process  demonstrates  this  is  frequently  the  point  where 
the  team  realizes  they  are  missing  sufficient  expertise  necessary  to  fully  analyze  the 
different  alternatives  proposed.  If  this  is  the  case,  the  SE  Facilitator  may  assist  in  helping  to 
obtain  the  necessary  information  or  expertise,  or  the  team  can  be  augmented  with 
appropriate  SME(s). 
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Step  4  is  envisioned  to  be  completed  in  one  or  two  team  meetings  with  necessary 
homework  completed  by  the  focal  points  for  each  potential  alternative  prior  to  and  in 
between  the  meetings.  At  the  first  meeting,  the  team  reviews  the  data  collected  for  each 
alternative  (Worksheet  3.2)  and  comes  to  agreement  as  to  the  validity  and  completeness  of 
the  information  provided.  As  part  of  this  same  meeting,  the  SE  Facilitator  can  conduct  an 
Alternatives  Evaluation  using  a  variety  of  Value  Analysis  methods.  Examples  of  the 
Worksheets  to  assess  Technology  and  Manufacturing  Readiness  Assessments  (Levels),  Risk 
Analysis  and  Value  Analysis  are  available  in  a  later  section  of  this  SE  Guide. 

Once  the  Alternative  Value  Analysis  is  complete,  the  Team  Leader  then  leads  the 
team  in  select  a  preferred  alternative,  or  alternatives.  Note,  for  many  S&T  programs, 
multiple  alternatives  may  be  chosen  for  further  technology  development,  with  the  highest 
payoff  approach  eventually  down-selected  at  higher  levels  of  technical  maturity.  Once  a 
preferred  alternative  is  selected,  greater  detailed  S&T  Program  Planning  information  can 
be  obtained  and  documented.  Caution  is  urged  to  review  the  results  of  the  value  analysis 
carefully;  does  the  team  agree  with  the  scoring  for  each  alternative,  and  are  there  errors  in 
assumptions  which  the  scoring  brings  to  light  which  need  to  be  corrected? 

Step  4,  Evaluation  of  Alternatives,  has  the  following  general  tasks: 

•  Agree  upon  an  alternative  scoring  system  and  ranking  methodology. 

•  Evaluate  (i.e.,  score)  how  each  alternative  is  expected  to  perform  with  respect  to 
each  S&T  exit  criteria  (threshold  and  objective)  using  Worksheet  4.1. 

•  Identify  and  qualitatively  estimate  the  risks  and  potential  risk  mitigation 
approaches  associated  with  each  alternative  actually  achieving  the  expected 
performance  with  respect  to  the  S&T  exit  criteria  using  Worksheet  4.2; 

•  Calculate  a  relative  risk  factor  for  each  alternative. 

•  Using  Worksheet  4.3,  assess  and  iterate  the  alternative  evaluation  results;  this  is 
known  as  Value  Analysis. 

•  Develop  the  team  consensus  on  preferred  alternative (s),  including  selection 
rationale;  document  on  Worksheet  4.3. 

In  the  S&T  environment,  by  its  very  nature,  so  much  is  unknown  that  these  tasks  can 
often  appear  to  be  both  difficult  and  frustrating.  The  first  time  a  team  works  the  process  is 
typically  the  most  difficult,  but  it  does  get  easier  as  the  process  proceeds.  The  process  itself 
is  not  difficult,  but  it  does  force  a  Systems  Engineering  discipline  on  a  team  that  will  help  it 
to  arrive  at  a  rational  preferred  solution. 
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During  this  evaluation  process,  S&T  exit  criteria  may  end  up  being  re-thought  and 
rephrased.  Some  may  be  dropped,  while  others  may  be  added.  There  may  be  missing 
information  that  is  important,  but  not  so  difficult  to  discover  and  include.  Usually,  at  this 
point  in  a  streamlined  SE  process,  high  precision  is  neither  possible  nor  necessary.  Critical 
issues  will  become  apparent  and  further  analysis  often  will  reveal  whether  they  justify 
additional  work.  The  proposed  alternatives  were  developed  by  the  team  with  an 
expectation  that  they  will  solve  the  problem  to  the  satisfaction  of  the  customer.  Evaluation 
of  Alternatives  is  the  step  where  those  expectations  are  initially  rationalized,  quantified 
and  ranked  to  produce  a  preferred  solution(s). 

Scoring  System  and  Ranking  Calculation 

The  team  is  free  to  select  a  scoring  and  ranking  system  of  its  own  design.  The 
simple  weighted  additive  system  described  in  Step  4  Worksheets  is  suitable  for  a  first  cut, 
streamlined  SE  process.  Higher  level  systems  such  as  those  based  on  weighted  geometric 
means  could  subsequently  be  implemented  if  desired. 

This  simple  methodology  has  the  following  characteristics: 

•  Relatively  intuitive  for  people  with  a  technical  background. 

•  Easily  implemented  in  the  attached  worksheets. 

•  Failing  to  meet  any  one  or  more  of  the  S&T  exit  criteria  thresholds  should  result  in  a 
zero  composite  score  so  that  excelling  in  meeting  one  S&T  exit  criteria  does  not 
mask  the  failure  to  meet  other  criteria. 

An  example  criteria  scoring  table  is  shown  below: 


Score  Value 

Description 

5 

Expected  to  meet  or  exceed  the  objective  S&T  exit  criteria 

4 

Expected  to  fall  between  the  threshold  and  objective  S&T  exit  criteria 

3 

Expected  to  meet  the  threshold  S&T  Exit  Criteria 

1 

Expected  to  fall  short  of  the  threshold  S&T  Exit  Criteria,  but  there’s 
some  hope  from  additional  development 

0 

Cannot  foreseeably  meet  the  S&T  Exit  Criteria 

•  Note:  This  method  does  not  incorporate  desireability  concepts  which  have  the 
advantage  of  dealing  more  powerfully  and  flexibly  with  non-linear  S&T  exit  criteria 
and  risk.  Desireability  concepts  require  special  software  and  a  leader  or  facilitator 
experienced  with  the  methodology. 
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Evaluate  each  Alternative  with  Respect  to  each  S&T  Exit  Criteria 

From  experience,  this  process  can  be  completed  most  efficiently  in  three  general 

tasks: 

1.  As  a  group,  evaluate  one  example  alternative  with  respect  to  each  S&T  exit  criteria 
and  risk,  preferably  led  by  someone  with  prior  process  experience. 

2.  Assign  qualified  S&T  Team  members  or  recruited  SME's  selected  alternatives  to 
evaluate  with  respect  to  each  S&T  exit  criteria  and  risk. 

3.  Again,  assemble  as  a  group  to  evaluate  and  edit  the  results  from  step  2. 

Task  1  can  sometimes  lead  to  refining  the  weighted  S&T  exit  criteria  generated 
during  step  two  of  the  Streamlined  SE  Planning  process.  These  useful  changes  are  best 
discovered  and  made  collectively  before  the  team  members  struggle  with  them 
individually.  It  is  helpful  to  consider  this  first  task  as  a  learning  experience,  but  once 
completed,  the  additional  tasks  often  require  only  a  fraction  of  the  time  of  the  first  effort. 

Often  at  this  point  of  the  process,  the  team  realizes  they  are  missing  sufficient 
expertise  or  information  necessary  to  fully  analyze  the  different  alternatives  proposed.  If 
this  is  the  case,  an  SE  Facilitator  may  be  able  to  step  in  and  help  to  obtain  the  necessary 
information  or  expertise,  or  the  team  can  be  augmented  with  appropriate  SME(s). 

Hopefully,  Step  4  can  be  completed  in  one  to  four  team  meetings,  with  necessary 
homework  completed  by  the  focal  points  for  each  potential  alternative  prior  to  and  after 
the  meetings.  It  is  rare  to  complete  this  step  in  one  meeting,  but  if  the  number  of  criteria 
are  less  than  10,  the  number  of  alternatives  are  fewer  than  five,  and  the  team  is  well 
acquainted  with  each  alternative,  it  can  be  accomplished. 

Team  Meeting  1: 

•  Team  agrees  upon  the  alternative  scoring  and  ranking  system. 

•  Team  reviews  the  data  collected  for  each  alternative  (Worksheets  3.2)  and  comes  to 

agreement  as  to  the  validity  and  completeness  of  the  information  provided. 

•  Team  scores  one  alternative  against  each  S&T  exit  criteria  as  a  working  exercise; 

need  to  refine  the  S&T  exit  criteria  is  determined. 

o  Evaluate  how  this  alternative  is  expected  to  perform  with  respect  to  each 
S&T  exit  criteria  (threshold  and  objective)  using  Worksheet  4.1. 

o  Identify  and  qualitatively  estimate  the  risk  associated  with  this  alternative 
actually  achieving  the  expected  performance  with  respect  to  the  S&T  exit 
criteria  using  Worksheet  4.2. 
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•  Assign  homework,  one  (or  occasionally  more  to  a  qualified  person)  alternative  to 
score  against  the  S&T  exit  criteria  and  risk  analysis. 

Team  Leader's  work  between  meetings 

•  Create  the  Value  Assessment  spreadsheet  (Worksheet  4.3). 

•  Collect  the  data  from  the  homework  assignments  and  enter  it  in  the 
spreadsheet. 

o  Based  on  the  S&T  Exit  Criteria,  calculate  a  relative  ranking  for  each 
alternative. 

o  Document  relative  risk  scores  for  each  alternative. 

Team  Meetings  2+: 

•  Assemble  and  review  the  results  of  the  individual  homework. 

•  Execute  the  Alternatives  Evaluation  using  the  agreed  upon  Value  Analysis 
methods  (the  team  leader  may  have  done  this  prior  to  the  meeting).  This  is 
essentially  populating  a  spreadsheet  with  the  agreed  upon  value  analysis 
method. 

At  this  point  people  usually  discover  specific  additional  information  is  useful  or 
necessary  for  a  valid  assessment. 

Value  Analysis  Evaluation 

The  results  of  the  value  analysis  will  be  a  relative  ranking  of  the  alternatives  based 

upon: 


•  Their  expected  ability  to  meet  the  S&T  exit  criteria,  and 

•  The  relative  risks  associated  with  each  alternative. 

The  team  leader  may  then  want  to  perform  at  least  a  rough  sensitivity  analysis  on 
the  results  (a  "what  if"  assessment).  The  question  to  ask  is  whether  small  changes  in  any 
alternative-to-exit  criteria  value  will  significantly  change  the  alternative  rankings?  The 
other  issue  to  explore  is  whether  the  relative  importance  of  the  S&T  exit  criteria  (weights) 
are  still  valid? 

Once  the  Alternative  Value  Analyses  are  complete,  the  Team  Leader  guides  the  team 
to  select  a  preferred  alternative,  or  set  of  alternatives.  In  many  S&T  programs,  several 
alternatives  are  often  selected  for  further  technology  development,  with  the  highest  payoff 
approach  eventually  down-selected  at  higher  levels  of  technical  maturity.  Once  a  preferred 
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alternative  (or  alternatives)  is  (are)  chosen,  a  decision  can  be  made  as  to  pursuing  more 
detailed  S&T  Program  Planning  information  for  their  proposed  solution  based  on  the 
results  of  the  Value  Analysis. 

Caution  is  urged  to  review  the  results  of  the  value  analysis  carefully;  does  the  team 
agree  with  the  scoring  for  each  alternative,  and  are  there  errors  in  assumptions,  which  the 
scoring  brings  to  light,  which  need  to  be  corrected? 

NOTE:  It  can  happen  that  a  given  planning  team  scores  the  alternatives  based  on 
their  level  of  knowledge  and  discussions,  only  to  have  alternatives  scores  modified  once 
new  SMEs  participate  and  contribute  their  knowledge,  often  in  the  areas  of  affordability 
and  manufacturability.  It  is  very  likely  the  planning  team  will  evaluate  the  scoring  more 
than  once,  conducting  further  research  into  the  alternative  technology,  and  adjusting  the 
scores  accordingly.  This  involves  an  iterative  process  of  discovery  and  does  not  mean  that 
one  person  or  the  team  as  a  whole  can  unfairly  or  intentionally  lead  the  planning  activity  to 
a  foregone  conclusion. 

At  the  conclusion  ofStep4,  the  team  should  have  a  strong  understanding  of  the 
rationale  for  the  preferred  alternative(s),  and  be  able  to  complete  the  elements  of  a  typical 
S&T  Plan,  in  the  format  specified  by  a  Technology  Review  or  other  format  as  required. 
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Step  4 


Evaluate 

Alternatives 


•  Assess  Alternatives 

-  Score  S&T 

Exit  Criteria 

-  Evaluate  Risk 

•  Select  Preferred 

Alternative(s) 


□  Value  Analysis 

-  Alternative  Scores 

-  Top  Risks  & 
Mitigation  Plans 

□  Rationale  for 
Preferred 

Alternative(s) 


Step  4  -  EVALUATE  ALTERNATIVES 

(See  Streamlined  SE  Process  Workbook  for  suggested  Worksheets] 


□  Worksheet  4.1  -  Desirability  for  each  S&T  Exit  Criteria 
Vs.  each  Alternative 

□  Worksheet  4.2  -  Risk  of  Achieving  each  S&T  Exit  Criteria 
Vs.  each  Alternative 

□  Worksheet  4.3  -  Composite  Scorecard 


Homework: 

The  Homework  is  to  complete  an  individual  perspective  of 
each  alternative  on  Worksheet  4.1.  . 
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□  Worksheet  4.1  -  Desirability  for  each  S&T  Exit  Criteria 

Vs.  each  Alternative 

To  conduct  a  complete  evaluation  of  alternatives  against  the  defined  Problem  Space, 
the  information  about  each  alternative  needs  to  be  collected  in  an  organized  format.  Use 
one  Worksheet  4.1  per  alternative  to  record  the  evaluation  against  each  of  the  agreed  to 
S&T  Exit  Criteria,  with  the  realization  that  additional  information  is  likely  to  be  generated 
as  the  team  expands  their  understanding  of  the  alternatives,  criteria,  and  the  problem. 


Alternative  Name: 

EC 

Exit 

EC 

■SB 

EC  Score 

Rationale 

ID 

Criteria 

Weight 

B9 

(Wtx  Sc) 

Alternative  Score  (EC1  Score  +  EC2  Score  +  EC3...) 

Alternative  Name:  Insert  the  name  of  the  alternative  solution. 

EC  ID:  Enter  S&T  Exit  Criteria  Identification  (tracking)  nomenclature  or  leave  blank 

Exit  Criteria:  Enter  S&T  Exit  Criteria  (EC)  name/title  from  Worksheet  2.5 

EC  Weight:  the  relative  importance  of  the  Exit  Criteria  compared  to  the  other  Exit 
Criteria.  Indicates  which  S&T  Exit  Criteria  are  more  important  that  others  to  solving  the 
problem. 

Alt  Score:  Score  as  to  how  well  the  alternative  satisfies  the  S&T  Exit  Criteria.  Team 
should  agree  upon  a  scoring  system  and  use  it  consistently  throughout  the  process. 

Examples  of  Scoring  Systems: 

Numerical  Scale:  0  to  5;  0  =  fails  to  5  =  meets/exceeds  objective  criteria 
+/-/0:  "+"  meets  threshold  criteria,  fails,  "0”  meets  current  capability 

EC  Score  (Wtx  Sc):  multiply  the  EC  Weight  value  times  the  Alt  Score. 

Alternative  Score  -  Add  each  EC  score  to  produce  the  overall  Alternative 
Desirability  Score.  If  three  S&T  Exit  Criteria  are  being  measured,  the  Alternative  Score 
would  be  EC1  Score  +  EC2  Score  +  EC3  Score. 

If  any  Alternative  fails  to  meet  one  or  more  S&T  Exit  Criteria  threshold  values,  the 
alternative  is  generally  accepted  as  undesirable  and  receives  a  total  composite  score 
of  zero  (0).  However,  it  is  still  recommended  the  Alternative  be  retained  for  the 
final  discussion  as  there  may  be  methods  to  overcome  the  failing  area(s). 

Rationale:  Use  this  area  to  document  the  logic,  both  Pro  and  Con,  used  in  the 
decisions  to  score  the  Weight,  Score,  and  EC  Score  values. 
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Step  4.  Evaluate  Alternatives  (Value  Analysis]  The  Measures  of  Merit 
are  Desirability  ( Worksheet  4.1)  and  Risk  ( Worksheet  4.2) 


Worksheet  4.1 
Desirability  for  each 
S&T  Exit  Criteria 
Vs.  each  Alternative 

(One  Worksheet  per  Alternative) 


Project  Name: 
Member  Name: . 

Role: . 

Worksheet  Date: 


Certain  Exit  Criteria  (EC)  are  more  important  than  other  Exit  Criteria.  Enter  an  EC  Weight 
value  from  Step  2  to  score  the  relative  importance  of  each  exit  criteria  compared  to  the  other 
exit  criteria. 


Certain  Alternatives  are  more  credible  to  attain  the  Exit  Criteria.  Enter  an  Alternative  (Ait) 
Score  as  to  how  well  the  Alternative  satisfies  each  S&T  Exit  Criterion  based  on  a  team  establish 
scoring  method.  If  any  Alternative  fails  to  meet  one  or  more  S&T  Exit  Criteria,  a  zero  score  is 
entered.  A  zero  score  essentially  fails  that  alternative  (EC  Wt  Score  x  0  -  0),  however,  it  is  still 
recommended  the  Alternative  be  retained  for  the  final  discussion. 

Multiply  EC  Weight  times  the  Alt  Score  (Wt  x  Sc)  for  an  "EC"  Score.  Add  ail  the  EC  scores 
together  for  an  overall  Alternative  Score  (EC1  Score  +  EC2  Score  +  EC3...) 


Alternate  Name: 

EC 

ID 

Exit 

Criteria 

EC 

Wt 

a 

EC  Score 

(Wtx  Sc) 

Rationale 

Alternative  Score  (EC1  Score  +  EC2  Score  +  EC3...) 

DOCUMENT:  The  Team  should  negotiate  one  overall  Alternative  score  for  each  Alternative 
that  is  then  transferred  and  compiled  on  the  Composite  Worksheet  4.3 

The  SE  " Conversation "  continues  '‘Desirability  is  a  S&T  Exit  Criteria  value  with  room  for  flexibility" 
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□  Worksheet  4.2  -  Risk  of  Achieving  each  S&T  Exit  Criterion 

Vs.  each  Alternative 

In  the  S&T  planning  context,  Risk  Assessment  is  the  determination  of  risk  related 
to  the  alternative’s  ability  to  achieve  the  threshold  values  of  the  S&T  Exit  Criteria.  In  Step 
4.2,  risks  are  determined  and  assessed  for  probability  of  occurrence  and  consequence. 

Steps  in  the  Risk  Assessment  process  include: 

□  Identify,  characterize,  and  assess  the  major  threats  to  achieving  the  S&T  Exit 
Criteria  threshold  values 

□  Identify  ways  to  reduce  and/or  eliminate  each  risk,  known  as  risk  mitigation 

□  Prioritize  risk  reduction  measures  based  on  the  technology  development  strategy 

Each  risk  should  be  assessed  for  probability  of  occurrence  and  potential  impact.  High 
Risk  simply  means  the  team  should  consider  Risk  vs.  Payoff  plus  Mitigation!  The  score 
should  also  consider  difficulty  of  risk  mitigation. 

On  worksheet  4.2,  the  Team  should  begin  by  identifying  the  top  risks,  along  with 
potential  risk  mitigation  approaches  for  each  Alternative  to  meet  the  threshold  values  for 
the  S&T  Exit  Criteria;  typically,  risks  can  be  associated  with  one  or  more  of  the  S&T  Exit 
Criteria.  The  Team  then  scores  the  Alternative  against  each  S&T  Exit  Criteria  from  a  risk 
standpoint  to  establish  a  Risk  Value  Score.  Each  potential  risk  should  be  scored,  based  both 
on  its  potential  impact  to  the  S&T  Exit  criteria,  as  well  as  to  the  ease  of  implementation  of 
risk  mitigation  approaches.  Following  the  evaluation  of  each  of  the  individual  risks,  the 
Team  should  negotiate  one  overall  Risk  score  for  each  Alternative  that  is  then  transferred 
to  Composite  Scorecard,  Worksheet  4.3. 

Alternative  Name:  Insert  the  name  of  the  alternative  solution. 

EC  ID:  Identification  code,  numerical  or  otherwise,  to  label  and  track  Exit  Criteria  (EC) 

Exit  Criteria:  Enter  S&T  Exit  Criteria  name/title  from  Worksheet  2.5 

Risk  Description:  Describe  the  Risk(s)  associated  with  each  alternative  actually  achieving 
the  expected  performance  with  respect  to  the  S&T  Exit  Criteria 

Risk  Score:  Calculate  a  score  based  on  a  team  agreed  upon  scoring  scale. 

Potential  /  Mitigation:  Describe  the  Potential  for  Risk  and  the  Mitigation  Plans  including 
any  difficulties  of  risk  mitigation. 
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Worksheet  4.2 
Risk  of  Achieving 
each  S&T  Exit  Criterion 
Vs.  each  Alternative 

(One  Worksheet  per  Alternative) 

The  Team  identifies  the  top  Risks  for  each  Alternative  to  meet 
the  threshold  values  for  all  of  the  S&T  Exit  Criteria. 

The  Team  evaluates  the  Risk  of  each  Alternative  against  each  S&T  Exit  Criteria 
and  generates  a  Risk  Score  (Risk  in  this  context  is  the  probability  of  failure  (Pf) 
to  exceed  the  Exit  Criterion  minimum  limits  (Worksheet  2.5)). 

High  Risk  does  not  mean  the  team  should  not  explore  all  possible  Alternatives 
( see  Payoff  to  Maturity  space  on  Worksheet  3.2).  High  Risk  simply  means  the  team 
should  consider  Risk  vs.  Payoff  plus  Mitigation  difficulty.  The  score 
should  also  include  consideration  of  mitigation  difficulty. 


Alternate  Name: 

EC 

ID 

Exit 

Criteria 

Risk 

Description 

Risk 

Score 

Potential  /  Mitigation 

Project  Name:  _ 
Member  Name:  _ 
Role: _ 

Worksheet  Date: 


Overall  Risk  Score:  _ 

DOCUMENT:  The  Team  should  negotiate  one  overall  Risk  score  for  each  Alternative.  Each 
Alternative’s  overall  Risk  Score  is  then  transferred  to  the  Composite  Scorecard  -- 
Worksheet  4.3 


...the  “Conversation"  continues. ..What's  the  Probability  of  Failure  (Risk)  to  achieve  each  S&T  Exit  Criterion? 
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□  Worksheet  4.3  -  Composite  Scorecard 

Worksheet  4.3  may  be  the  last  worksheet  of  Step  4,  but  it  is  certainly  not  the  end  of  the 
SE  Conversation.  Once  all  the  Alternative  Desirability  and  Risk  scores  have  been  entered 
onto  the  Composite  Scorecard,  the  team’s  next  task  is  to  step  back,  analyze  the  results,  and 
select  a  preferred  alternative,  or  alternatives. 

EC  ID:  Enter  any  S&T  Exit  Criteria  tracking  nomenclature  or  leave  blank 

Exit  Criteria:  Enter  S&T  Exit  Criteria  name/title  from  Worksheet  2.5 

Alternative  Name:  Insert  the  name  of  the  alternative  solution  or  other  identifier 

Composite  Score:  Transfer  Alternative  Score  from  Worksheet  4.1 

Risk  Score:  Transfer  Alternative  Risk  Score  from  Worksheet  4.2 

Preferred  Alternative  (s)  /Rationale:  Indicate  the  Team’s  selection 

Preferred  Alternative(s)  Decision  Documentation:  Include  any  documentation 
arguments  justifying  the  selection  of  the  preferred  alternative 


The  Composite  Scorecard  is  used  to  help  guide  the  planning  team  to  a  Preferred 
Alternative(s).  The  scores  on  this  worksheet  should  be  analyzed  by  asking  the  following 
questions: 

1  -  "Does  the  ranking  of  the  composite  scores  pass  the  sanity  test,  i.e.,  do  the  values 
make  sense?"  If  the  scores  of  more  than  one  alternative  are  very  close  in  total  value, 
how  do  the  distinguishing  strengths  and  weaknesses  of  the  competing  alternatives 
impact  solving  the  problem  at  hand? 

2  -  "Is  risk  level,  taking  into  account  potential  risk  mitigation  approaches,  acceptable 
for  the  higher  scoring  Alternatives?” 

Based  on  this  analysis,  the  team  normally  selects  its  Preferred  Alternative(s)  and 
moves  on  to  the  Step  5  Action  Plan  to  fully  develop  the  Preferred  Alternative(s). 
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Worksheet  4.3 
Composite  Scorecard 


Project  Name: 
Member  Name: . 

Role: . 
Worksheet  Date: 


Transfer  all  Alternative  and  Risk  scores  from  each  Alternatives  Worksheet  ( 4.1  and  4.2) 

to  this  Composite  Scorecard 


ALTERNATIVES 

EC 

ID 

Exit  Criteria 

Alternative 

Alternative 

Alternative 

Alternative 

Alternative  Score 

(Worksheet  4.1) 

Risk  Score 

(Worksheet  4.2) 

Preferred  Alternative  (s) 

/  Rationale 

Preferred  Alternative  (s) 
Decision  Documentation 

The  Composite  Scorecard  is  not  the  final  answer,  but  presents  the  team  with  values  for 
them  to  agree  on  the  Preferred  Alternative (s). 

Analyze  the  scores  by  asking  the  questions: 

1  -  Does  the  ranking  of  the  composite  scores  pass  the  sanity  test,  i.e.,  do  the  values  make  sense? 

2  -  Do  Risks  outweigh  the  scores  for  the  high  scoring  Alternatives? 

Based  on  this  review,  the  team  moves  on  to  the  Action  Plan  for  the  Preferred  Alternative (s). 
Often  in  S&T  it  is  common  to  have  multiple  alternatives  selected  for  initial  development. 


...the  “Conversation”  continues. ..If  a  score  doesn't  make  sense  ...discuss  why  it  scored  as  it  did? 
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STEP  5  -  DELIVER  S&T  PLAN 


Step  1  Step  2  Step  3  Step  4  Step  5 


Form 

Determine  _ ^  Generate 

Evaluate  ^ 

Deliver 

Team 

Requirements  Alternatives 

Alternatives 

S&T  Plan 

Do: 

•  Finalize  AF  Problem 
/  Goals  l  Preferred 
Solubon(s) 

•  Identify  All 

•  Understand  Relevant 

Technologies 

'  -  Score  S&t'^  ' 

•  Establish  Planning 
Team 

•  Define  S&T 

ExitCrrtena 

•  Generate  Alternative 
Solutions 

-  Evaluate  Risk 

•  Prepare  for  Intended 
Action  Course 

a  idate  Customei 

Document: 

□  Problem  Definition 

□  Requirement  Set 
-  Performance 

□  Full  Description  of 
each  Alternative. 

□  Value  Analysis 
-  Alternative  Scores 

□  AF  Problem  /  Goal 

□  Preferred  Solubon(s) 

(includes  roles  & 

■  Affordability 

-  Top  Risks  & 

-  Roadmap 

U  Team  Charter 

•  Supportability 

□  Tech  Challenge  List 

□  S&T  Exit  Criteria  Set 

•  MRL  Levels 

M  itiqation  Plans 
□  Rationale  for 

Preferred 

Altematrve(s) 

□  Acbon  Plan 

Figure  10:  STEP  5  -  DELIVER  S&T  PLAN 


Step  5,  the  culmination  of  the  SE-based  planning  process,  delivers  a  Plan  for  the 
proposed  S&T  program  using  the  Preferred  Alternative (s)  selected  in  Step  4.  Often,  once  a 
preferred  alternative  is  selected  on  a  first  run  through  this  planning  process,  it  is  necessary 
to  continue  to  add  a  higher  level  of  detail  to  the  proposed  program  plan.  Additional  details 
on  topics  such  as  specific  program  technical  and  schedule  milestones  and  resources 
(funding,  manpower,  and  in-house  laboratory/test  facilities)  required  will  need  to  be 
obtained;  this  information  can  then  be  provided  to  the  appropriate  S&T  management  chain 
in  a  format  suitable  for  the  particular  program  advocacy  forum  (examples  include 
Technology  Investment  Reviews,  Technical  Review  Boards,  etc).  A  S&T  program  roadmap 
showing  these  milestones  (including  TRL  and  MRL  values)  and  required  funding  as  a 
function  of  fiscal  year  is  also  highly  beneficial.  For  relatively  high  TRL  programs,  additional 
detail  on  the  benefits  of  the  proposed  program  to  the  customer  using  the  Preferred 
Alternative  are  often  also  necessary. 

Assuming  the  team  does  arrive  at  a  Preferred  Alternative  (i.e.,  Technology  Solution) 
that  adequately  meets  the  S&T  Exit  Criteria  (and  the  foundational  customer  requirements), 
their  main  job  now  is  to  prepare  the  actual  program  plan  and  advocate  for  program 
approval/initiation. 
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Step  5 


Deliver 
S&T  Plan 


•  Finalize  AF  Problem 
/  Goals  /  Preferred 

Solution(s) 

•  Prepare  for  Intended 

Action  Course 


□  AF  Problem  /  Goal 

□  Preferred  Solution(s) 
-  Roadmap 

□  Action  Plan 


Step  5  -  DELIVER  S&T  PLAN 

( See  Streamlined  SE  Process  Workbook  for  suggested  Worksheets] 


□  Worksheet  5.1  -  Program  Action  Plan 

□  Worksheet  5.2  -  SE  Case  Study 


Homework: 

Before  the  team  meets  for  what  may  well  be  the  final  time 
during  the  planning  process,  thought  should  be  given  as  to 
the  best  approach  for  advocating  the  proposed  program  up 
the  S&T  management  chain.  This  may  well  include 
obtaining  additional  information  on  the  proposed  program 
and  creating  draft  documents  (roadmaps,  investment 
strategy  sheets,  etc)  that  will  be  required  for  program 
advocacy. 
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□  Worksheet  5.1  -  Program  Action  Plan 

The  Team  should  agree  upon  the  plan  to  advocate  for  approval/implementation  of  the 
proposed  S&T  program  incorporating  the  Preferred  Alternative (sj.  Depending  on  the  nature 
of  the  proposed  program,  the  Action  Plan  could  range  from  a  fairly  simple  White  Paper  or 
Roadmap  to  a  fully  detailed  advocacy  briefing/package.  Worksheet  5.1  provides  a  possible 
outline  based  on  principles  of  good  program  management. 

Details  necessary  to  complete  this  Action  Plan  include  programmatic  technology 
performance,  costand  schedule  estimates,  a  basic  program  execution  strategy  of  external 
contracts  and  in-house  activities,  as  well  as  the  type  of  funding  (6.1,  6.2,  6.3)  required. 
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Step  5.  Build  the  Plan:  With  the  Problem  Space  understood  and  the  Solution  Space  fully 
explored,  Advocacy  Briefs  and  Action  Plans  can  be  generated  for  the  Preferred  Alternative(s). 

Project  Name: _ 

Worksheet  5.1  Member  Name: _ 

Program  Action  Plan  Role: _ 

Worksheet  Date: _ 

The  Team  should  agree  upon  the  plan  to  advocate  for  approval/implementation  of  the 
proposed  S&T  program  incorporating  the  Preferred  Alternative (s).  Depending  on  the  nature 
of  the  proposed  program,  the  Action  Plan  could  range  from  a  simple  White  Paper  or  Roadmap 
to  a  fully  detailed  advocacy  briefing/package.  Worksheet  5.1  provides  a  possible  outline 
based  on  principles  of  good  program  management. 

Details  necessary  to  complete  this  Action  Plan  include  programmatic  technology 
performance,  costand  schedule  estimates,  a  basic  program  execution  strategy  of  external 
contracts  and  in-house  activities,  as  well  as  the  type  of  funding  (6.1,  6.2,  6.3)  required. 


Program  Action  Plan  Checklist  (possible  outline) 

□  Document  details  of  the  planning  process: 

□  AF  problem 

□  Customer  (s)/owners/end  users 

□  Requirements 

□  S&T  Exit  Criteria 

□  Details  of  each  alternative  solution  considered 

□  Preferred  alternative (s)  selected  (include  rationale) 

□  Document  proposed  program  execution/management  plan 

□  Proposed  S&T  program  schedule  (with  milestones  and  decision  points)  -  may  be 
in  the  form  of  a  technology  roadmap 

□  Required  resources 

□  Proposed  execution  approach  (inhouse,  external  contracts) 

□  Risk  management  approach 

•  Describe  the  recommended  next  steps  in  advocacy  process: 


...the  SE  “Conversation"  never  stops.. .A  well  thought  out  plan  is  easier  to  defend  and  execute. 
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□  Worksheet  5.2  -  SE  Case  Study 


The  Case  Study  summarizes  how  the"  use  of  the  Streamlined  S&T  Process  helped  or  didn’t 
without  finger  pointing. ..Another  words,  the  lessons  learned  of  the  process:  Was  the  right 
problem  identified  correctly?  Were  the  right  requirements  derived  from  the  problem? 
Were  the  stakeholders  who  needed  to  be  on  the  team  invited  and  participated?  Etc. 

It  is  with  hope  the  process  of  identifying  the  problem,  defining  the  requirements,  and 
generating  alternatives,  and  the  manner  by  which  the  team  came  to  select  the  Preferred 
Alternative,  was  a  positive  experience.  Other  projects  who  have  used  this  process,  commented, 
the  learning  journey  was  very  rewarding  as  the  knowledge  and  preconceived  notions  changed 
as  the  SE  Conversation  continued. 


The  Case  Study  is  a  high-level  account  of  the  team’s  work  to  arrive  at  a  decision 
recommending  a  preferred  solution  and  action  plan.  Did  you  have  appropriate  Customer 
involvement?  Was  the  Problem  Statement  captured  accurately?  What  lessons  did  the  team 
learn  in  the  process  of  determining  a  solution?  The  Case  Study  is  a  place  to  summarize  the 
positive  and  the  negative  without” finger  pointing"  anyone  specific,  but  a  constructive  report 
that  can  be  filed  on  the  SE  CoP  website  for  others  to  learn  from. 

The  outline  provided  in  Worksheet  5.2  is  only  notional  and  can  be  whatever  the  team/author 
decides  to  report. 
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Worksheet  5.2 
SE  Case  Study 


While  the  actions ;  activities,  and  experiences  of  applying  the  Systems  Engineering 
Streamlined  Planning  Process  are  still  fresh  in  the  mind  of  the  Core  Team  and  other 
participants,  a  Case  Study  is  a  useful  document  to  capture  the  team’s  discoveries. 

Case  Study 

Description  of  Problem  Space 

Background 
Study  Objectives 

Study  Process 

Description  of  Technical  Effort 
The  Integrated  Product  Team 
Kickoff  Meeting,  date 

Technical  and  Systems  Engineering  Process  Conclusions 

Systems  Engineering 

Additional  Observations 

Lessons  Learned  and  Recommendations 

Appendix  A.  List  of  Acronyms  and  Terms 
Appendix  B.  References 


...the  SE  “Conversation"  never  stops. ..don't  be  surprised  to  find  this  process  wasn't  worth  the  time  invested. 
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EXAMPLE  PROGRAM:  Aerospace  Ground  Equipment  Lifetime 
Coating  Evaluation  for  System  Sustainment  (AGELESS) 


Problem  Background: 

The  Air  Force  annually  spends  over  $69M  annually  on  corrosion  mitigation  associated  with 
non-powered  support  equipment,  primarily  portable  aircraft  work-stands.  Labor  and  material 
costs  are  increasing,  while  workforce  shaping  is  reducing  available  maintenance  resources. 

Study  Objectives: 

This  FY07-08  AFRL/RX  Director-sponsored  Company  Grade  Officer  Initiative  Program,  a 
“quick-look"  mini-program  with  direct  field  application,  used  streamlined  SE  methods  to  select  an 
optimum  method  for  corrosion  mitigation  of  non-powered  AGE.  A  number  of  technology 
alternatives  were  considered,  including: 

•  Coatings  conforming  to  MIL-PRF-26925 

•  Metallization  (aka  metal  wire  arc  spray] 

•  Hot-dip  galvanization 

•  Electro  galvanization 

These  four  potential  solutions  will  be  used  in  the  example  worksheets  that  follow;  it  should  be 
noted  that  although  each  of  these  solutions  is  much  more  mature  than  technologies  usually 
considered  for  an  S&T  program  plan,  they  still  serve  a  useful  purpose  in  illustrating  the  Streamlined 
SE  Planning  method  for  S&T. 

In  this  study,  the  hot-dip  galvanization  process  was  determined  to  be  the  best  solution 
because  it  possesses  superior  capabilities  for  three  high  priority  requirements  and  no  inferior 
capabilities  with  respect  to  the  remaining  requirements.  Hot-dip  galvanization  is  applicable  to  only 
low  strength  steels,  which  are  the  materials  used  in  aerospace  ground  equipment  structures.  The 
hot-dip  galvanization  process  was  assessed  to  have  superior  capabilities  to  treat  hard-to-reach 
surfaces  and  was  the  preferred  alternative  by  USAF  maintainers. 

The  electro  galvanization  process  was  determined  to  possess  superior  capabilities  for  two 
high  priority  requirements,  but  had  issues  with  widespread  availability  and  limitations  for  treating 
hard-to-reach  surfaces.  However,  this  process  is  not  limited  to  low  strength  steels  and  could  be  the 
preferred  choice  for  coating  other  applications.  The  metallization  process  has  superior 
capabilities  for  reducing  environmental  impact  and  for  producing  damage  tolerant  protection,  but  it 
has  major  issues  in  hard-to-reach  surfaces  and  is  also  too  expensive,  thus  failing  to  meet  threshold 
requirements.  Additionally,  there  were  some  concerns  the  metallization  process  would  change 
metallurgical  properties  of  the  structure,  which  is  undesirable.  The  coatings  conforming  to  MIL- 
PRF-26915  provided  essentially  the  same  capabilities  as  the  current  painting  practices,  with  some 
advantages  of  producing  more  damage  tolerant  protection  and  but  with  some  disadvantages  of 
requiring  extra  processing  steps. 

Results  of  this  study  were  presented  to  the  Aircraft  Ground  Support  Equipment  Working  Group 
(AGSEWG],  with  final  recommendations  made  to  the  Combat  Sustainment  Group  for  analysis. 
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Worksheet  Examples 


The  following  Example  Worksheets  are  based  on  an  actual  Aerospace  Ground 
Equipment  Longevity  Coating  Evaluation  for  System  Sustainment  (AGELESS)  Study 
performed  by  an  RX  CGO  SE  Team.  The  general  results  of  this  study  can  be  used  to 
demonstrate  the  possible  results  of  a  team  using  the  Streamlined  S&T  Planning  Process 
develop  a  recommended  program  plan. 
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Step  1:  Form  Team.  Start  to  think  of  the  team  in  specific  terms  of  the  problem  at  hand. 
Different  problems  require  different  members  with  their  experiences  and  skills. 


Worksheet  1.1 
Define  Problem 
EXAMPLE 


Project  Name:  AGE  Corrosion  Mitigation 

Member  Name: _ lack  Sparrow _ 

Role:  Team  Lead/Program  Manager 

Worksheet  Date: _ Today _ 


The  team  works  with  the  customer(s)  to  scope  the  problem  space. 

Who  is  the  Customer?  [Who  brought  the  problem?)  AGE  Program  Office  at  WR- 
ALC/LEEV 

Time  frame  of  Problem  (When  Needed/Urgency ?):  _ Near-Term _ 

Describe  the  Problem  (Scope/Major  Issues/Constraints):  Corrosion  mitigation  costs 
too  high  for  non-powered  Aerospace  Ground  Equipment  fAGEl!  Annual  $69M+ 

corrosion  mitigation  costs,  with  increasing  labor  &  material  costs:  workforce 

shaping  has  reduced  maintenance  man  hours  fMMHl  available.  Desire:  extended 

coating  life,  not  total  AGE  life. 

Who  is  the  Problem  Owner?  AGE  Program  Office  at  WR-ALC/LEEV _ 

Who  is  the  End  User?  (Who  turns  the  wrench/pushes  the  button?)  All  AF  Flight-Line  Ops 

Why  Should  AFRL/RX  be  Working  This  Issue?  rx.  through  rxs  Coating  Tech 
Integration  Office  and  Corrosion  Prevention  &  Control  Office,  has  core  tech  competency  in 

this  area. 


Who  Needs  to  be  Included  as  Partners  In  This  Issue? 


What  Do  We  NOT  Know?  What  are  the  specific  "bad  actors,"  which  drive  AGE 
coating  problemfsl:  how  often  &  how  are  current  coating  repairs  made? 


...  the  "SE  Conversation"  begins  with  understanding  of  the  problem.. .What  is  the  "Discontent?”  ...and  Why? 
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Worksheet  1.2 
Team  Make-up  /  Roles 

EXAMPLE 


Project  Name:  AGE  Corrosion  Mitigation 

Member  Name: _ lack  Sparrow _ 

Role:  Team  Lead/Program  Manager 
Worksheet  Date: _ Today _ 


With  the  Problem  understood,  a  team  approach  ( Core  plus  Augmentees) 
must  by  defined,  i.e.,  identify  the  skills  needed  to  discuss  and  refine  the  Problem 

Team  Directory,  Roles  &  Responsibilities 


Role  /Resoonsibilitv 

ORG/Svmbol 

Name  /  Contact  Info 

CORE  TEAM  MEMBERS 

•  Team  Lead  /  PM 

AFRL/RXS 

lack  Snarrow 

•  Princinal  Investigator 

CTIO  RA 

Mike  Snicer  fRXSSl  Chief  Eng 

•  In-House  Researcher 

RXSA  UDRI 

A1  Topcoat 

•  Corrosion  Engineer 

AFCPCO/WR-ALC 

Rustv  Standish 

•  SE  Facilitator 

AFRL/RXOB 

Dr.  lim  Malas 

•  Customerfsl  fAcquirerl 

WR-ALC/LEEV 

Mr.  David  Robert 

•  Customerfsl  fUser  1 

445  AW/A4  fC-51 

MSgt  Albert 

Augmentees  (includes  SMEs,  Finance,  Contracting,  etc.) 

•  Finance 

AFRL/RXF 

Ms.  Paula  Monevnennv 

•  Contracting 

AFRL/RXK 

Mr.  Tv  Downe 

• 

• 

...the  "SE  Conversation"  continues...  including  all  the  right  expertise  and  Stakeholders 
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Worksheet  1.3 
Establish  Team 
Charter 
EXAMPLE 


Project  Name:  AGE  Corrosion  Mitigation 

Member  Name: _ lack  Sparrow _ 

Role:  Team  Lead/Program  Manager 

Worksheet  Date: _ Today _ 


Core  Team  stays  with  the  entire  process.  Other  persons 
with  needed  expertise  are  brought  in  only  where  appropriate. 

Use  Worksheets  1.1  and  1.2  as  backup  to  this  activity 

“Planning”  Team  Charter 

Goal/Objectives:  Plan  S&T  program  to  address  increasing  non-powered  AGE 

coating  corrosion  mitigation  costs 

•  Schedule 

o  Meeting  frequency  (Weekly,  monthly,  etc.)  Meet  once  per  week _ 

O  Duration:  start :  tmm/dd/vvvvl  03/19/2007  End:  tmm/dd/vvvvl  09/30/2008 

•  Resources  (Available  to  the  Team):  normal  office  support  &  TDY  funding  for  two 
face-to-face  meetings 

•  Authority  /  Accountability  of  Team  Membership:  Dr.  H.  Honcho,  RXS  Div 
Chief 

As  a  Core  Team  Member  for  this  planning  effort,  I  understand 
the  following  is  expected  of  all  Core  Team  Members: 

o  Commit  to  participate  in  all  team  meetings 
o  Commit  to  complete  all  "homework”  on  schedule 

o  Participate  in  the  documentation  of  each  step  before  proceeding  to  the 
next  step 


Signatures  of  all  core  planning  team  members  commitment 


1. 

4. 

2. 

5. 

3. 

6. 

(The  above  is  just  a  suggested  outline.  Expand  with  extra  pages  if  needed) 


...  the  SE  Conversation  is  solidified  when  a  Team  Charter  formalizes  expectations 
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Step  2:  Requirements.  A  firm  understanding  of  the  requirements,  technical  challenges,  and 
how  they’re  stated  as  S&T  Exit  Criteria  are  critical  steps  in  Problem  Space  documentation. 

Project  Name:  AGE  Corrosion  Mitigation 

Member  Name: _ lack  Sparrow _ 

Role:  Team  Lead/Program  Manager 

Worksheet  Date: _ Today _ 

The  Core  Team  refines  the  understanding  of  the  Customer’s  Problem 
into  stated  requirements  and  validates  them  with  the  Customer 


Worksheet  2.1 
List  of  Customer 
Requirements 
EXAMPLE 


Requirement 

Name 

Requirement  Description 

(as  specific  as  possible] 

Threshold* 

Objective* 

Coating 

Durability 

New  coating  solution  with  at  least  2X 
lifetime  over  current;  Desire  4X  current 
(Between  PDM) 

2X 

4X 

EPA  /  ESH 
Compliant 

Reduce  environmental  waste  stream 

30%  Less 

50%  Less 

Coating 

Coverage 

Ability  to  coat  /  treat  "Hard  to  Reach” 
surfaces  (inside  4ft  long,  1  inch  ID  pipe) 

95% 

100% 

Affordability 

Reduction  in  corrosion  mitigation 
materials  costs  and  labor 
(funding  &  manpower) 

40% 

reduction 

60% 

reduction 

Ease  of  Repair 

Reparability  at  least  as  good  as  current 

Same 

better 

Looks / 
Appearance 

Clean  appearance 

*Desirable  -  vary  with  maturity  of  task 


Threshold:  The  measurable  value  that  must  be  achieved  for  the  technology  to  advance  to  the  next 
stage  of  development  or  transition  (usually  given  as  a  minimum  or  maximum  value]. 

Objective:  The  stretch-goal  value  that  is  desirable,  but  not  essential  (often  viewed  as  the  requirements 
trade  space  if  parameters  such  as  cost  or  schedule  are  more  important  to  the  customer]. 

□  DOCUMENT:  Validate  this  worksheet  with  your  customer  representatives /stakeholders 
to  ensure  agreement  with  the  goals. 


...the  “Conversation”  continues...  What  capability  does  my  Customer  (Want]  Need?. ...and  Why? 
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Project  Name:  AGE  Corrosion  Mitigation 

Member  Name:  lack  Sparrow _ 

Role:  Team  Lead/Program  Manager 

Worksheet  Date: _ Today _ 

The  Team  now  has  to  turn  the  Customer  Requirements  into  actionable 
S&T  descriptions  S&T  Exit  Criteria).  This  step  helps  the  team  decompose 
the  Customer  Requirements  into  Technical  Issues  (Challenges). 


Worksheet  2.2 
Technical  Challenges 
EXAMPLE 


Requirement  Name 

Technical  Challenge  Issues 

Coating 

Durability 

Coating  adhesion  to  base  metal  structure  with  no  change  to 
metallurgical  structure,  while  exhibiting  desired  durability 
characteristics 

EPA  /  ESH  Compliant 

Coating  applied  in  a  non-toxic,  low  hazardous  waste  process; 
Green/earth  friendly 

Coating 

Ability  to  coat  total  surface  including  recessed  and  tight 

Coverage 

access  areas,  and  inside  diameters  of  open  tubing/pipe 

Affordability 

Coating  process  and  materials  affordable 

Ease  of  Repair 

Durable  Repairs  within  currently  accepted  repair  times 

....the  SE  Conversation  continues  by  identifying  "Where’s  the  S&T  in  this  challenge?". ..and  Why? 
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Worksheet  2.3 
List  of  S&T 
Exit  Criteria 
EXAMPLE 


Project  Name:  AGE  Corrosion  Mitigation 

Member  Name: _ lack  Sparrow _ 

Role:  Team  Lead/Program  Manager 

Worksheet  Date: _ Today _ 


Customer  Requirements  are  usually  at  fairly  high  level.  The  team  now  has  to  define  and 
document  what  a  final  product,  technology,  or  system  must  do,  including  the  parameters 
that  define  successful  completion.  Worksheets  2.1  and  2.2  should  lead  into  this  step. 


Criteria  Name 

Description 

Threshold* 

Objective* 

RQT  Name 

Coating 

Durability 

Resistance  to  impact  & 
weather  (rust/corrosion] 

2X  current 

4X  current 

Coating  Durability 

EPA  /  ESH 
Compliance 

Reduce  environmental 

waste  stream 

30% 

Reduction 

40% 

Reduction 

EPA  /ESH 
Compliant 

Surface  Coverage 

Amount  of  surface 
covered 

95% 

100% 

Coating  Coverage 

Affordability 

Reduction  in  corrosion 
mitigation  materials  costs 
and  labor 

40% 

Reduction 

60% 

Reduction 

Affordability 

Repairability 

Ease  of  repair 

Same  as 

current 

Better  than 

current 

Repairability 

*Desirable  -  vary  with  maturity  of  task 


Threshold:  The  measurable  value  that  must  be  achieved  for  the  technology  to  advance  to  the  next 
stage  of  development  or  transition  (usually  given  as  a  minimum  or  maximum  value). 

Objective:  The  stretch-goal  value  that  is  desirable,  but  not  essential  (often  viewed  as  the 
requirements  trade  space  if  parameters  such  as  cost  or  schedule  are  more  important  to  the  customer). 


...the  “Conversation”  continues...  What  S&T  Exit  Criteria  demonstrate  the  Requirements )  have  been  met?"  - 

....and  Why? 
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Worksheet  2.4 
Group  S&T 
Exit  Criteria 
by  Category 
EXAMPLE 


Project  Name:  AGE  Corrosion  Mitigation 

Member  Name: _ lack  Sparrow _ 

Role:  Team  Lead/Program  Manager 


Worksheet  Date: _ Today 


S&T  Exit  Criteria  negotiated  by  the  team  can  usually  be  grouped  into  common  topics, 
identified  such  as  P-1,  P-2,  etc.  for  Performance,  C-l,  C-2,  etc.  for  Cost. 

There  is  no  limit  to  the  number  of  S&T  Exit  Criterion  but  this  is  the  logical  place  to  consider 
consolidation,  combining  the  criteria,  if  appropriate. 


ID 

S&T  Exit  Criteria 

Description 

Group:  PERFORMANCE 

P-1 

Durability 

Resistance  to  impact  &  weather 

P-2 

EPA  /  ESH 
Compliance 

Reduce  environmental  waste  stream 

P-3 

Surface  Coverage 

%  of  total  surface  area  covered 

P-4 

Repairability 

Ease  of  repair 

Group:  COST 

C-l 

Affordability 

Reduction  in  corrosion  mitigation  materials  costs  and  labor 

Possible  Category  Groupings  (add  categories  depending  on  technology) 


C=  Cost  (development,  acquisition,  deploy,  point  of  use  operation) 
S  =  Schedule 

P  =  Performance  (reliability,  weight,  footprint,  set-up  time) 

L  =  Logistics  (transport,  service  life,  storage,  scalability,  disposal 


HF  =  Human  Factors 

E  =  Environment  (EPA,  certifications,  etc.) 

ST  =  Strategic  (outside  influences,  be  they  political  or  otherwise) 
POL  =  Political  (constrictions) 


...the  "Conversation”  continues...  What  groups  form  the  S&T  Exit  Criterion?” -....and  Why? 
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Worksheet  2.5 
S&T  Exit  Criteria 
Complete  Description 
EXAMPLE 


Project  Name:  AGE  Corrosion  Mitigation 

Member  Name: _ lack  Sparrow _ 

Role:  Team  Lead/Program  Manager 

Worksheet  Date: _ Today _ 


The  Team  defines  the  detailed  quantitative  attributes  of  the  S&T  Exit  Criteria. 


ID 

Name 

Priority 

/Wt* 

Units 

Threshold 

Value 

Threshold 

Rationale 

Objective 

Value 

Objective 

Rationale 

How 

Measured 

Group  PERFORMANCE 

p  l 

Durability 

1.0 

months 

2x 

Current 

Customer 

req’t 

4X  Current 

Customer 

goal 

Env  chamber 

tests 

P  2 

EPA  /  ESH 
Compliance 

0.7 

Gallons 

waste 

30% 

Reduction 

Customer 

req’t 

40% 

Reduction 

Customer 

goal 

Pilot  process 
tests 

P-3 

Surface 

Coverage 

1.0 

% 

surface 

covered 

95% 

Hand 
touch-up 
other  5% 

100% 

Customer 

goal 

Pilot  process 
tests 

P-4 

Repairability 

0.7 

Minutes 

per 

repair 

Same  as 

current 

Customer 

req’t 

Better 

than 

Current 

Customer 

goal 

Pilot  process 
tests 

Group  COST 

C-l 

Affordability 

1.0 

dollars 

40% 

reduction 

Customer 

req’t 

60% 

Reduction 

Customer 

goal 

Detailed  cost 
estimates 

*Can  be  used  to  compare  relative  importance  of  S&T  Exit  Criteria.  These  are  suggestive,  use  whatever  scale  you 
determine  relevant/needed.” 

Priority  qualitative  ranking:  High,  Medium,  Low. ...Or, 

Weight  scores  contribution  importance:  1  =  critical  contribution  to  capability,  0.7  important  contribution,  and 
0.5  if  not  too  important 

Threshold:  The  measurable  value  that  must  be  achieved  for  the  technology  to  advance  to  the  next  stage  of 
development  or  transition  (usually  given  as  a  minimum  or  maximum  value). 

Objective:  The  stretch-goal  value  that  is  desirable,  but  not  essential  (often  viewed  as  the  requirements  trade 
space  if  parameters  such  as  cost  or  schedule  are  more  important  to  the  customer). 


...the  "Conversation"  continues...  When  is  ‘more  or  less'  better  and  how  do  you  measure  it?” -....and  Why? 
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Step  3:  Generate  Alternatives.  Explore  and  list  all  the  possible  ways  to  solve  a  problem 
are  important  steps  in  documenting  the  Solution  Space. 


Worksheet  3.1 
List  Alternatives 
EXAMPLE 


Project  Name:  AGE  Corrosion  Mitigation 

Member  Name: _ lack  Sparrow _ 

Role:  Team  Lead/Program  Manager 

Worksheet  Date: _ Today _ 


The  Team  identifies  possible  solutions  ( Alternatives )  to  satisfy 
each  of  the  S&T  Exit  Criteria.  The  Alternatives  may  be  existing  technologies, 
but  the  team  should  also  explore  novel  approaches  that  may  offer 
enhanced  payoff...  even  if  it  seems  a  higher  risk. 


Alternative  Name 

Description 

Coatings  conforming 
to  MIL-PRF-26915 

New  paint  that  conforms  to  MIL-PRF-26915,  similar  to  current  paint 
methods.  However,  it  is  believed  there  are  additional  application 
process  steps  required. 

Metallization  (aka 
Metal  Wire  Arc 
Spray) 

Process  applies  a  thermally  sprayed  coating  of  metal  much  like 
spraying  paint.  Two  metal  wires,  zinc  or  aluminum,  are  fed  to  a  spray 
gun  device.  A  positively  charged  wire  meets  a  second  negatively 
charged  wire  at  the  gun  head,  creating  a  high  temperature  arc.  Dry 
compressed  air  atomizes  the  molten  material  and  propels  it  to  the 
surface  being  coated. 

Electro 

Galvanization 

Electro-plating  deposition  of  zinc  over  steel  or  iron  to  prevent  galvanic 
corrosion  of  the  underlying  surface.  Galvanizing  provides  the  relative 
corrosion  resistance  of  zinc,  which,  under  most  service  conditions,  is 
considerably  less  than  those  of  iron  and  steel.  The  zinc  is  consumed 
first  as  a  sacrificial  anode,  so  it  cathodically  protects  exposed  steel.  In 
any  scratches  through  the  zinc  coating,  the  exposed  steel  will  be 
cathodically  protected  by  the  surrounding  zinc  coating,  unlike  an  item 
which  is  painted  with  no  prior  galvanizing. 

Hot-Dip 

Galvanization 

Process  of  coating  iron,  steel,  or  aluminum  surfaces  with  a  thin  zinc 
layer  by  dipping  the  metal  in  a  molten  bath  of  zinc  at  860°F  (460°C).  A 
dull,  gray,  fairly  strong  sacrificial  zinc  coating  inhibits  corrosion, 
protecting  the  underlying  surface  from  the  elements.  50  microns  thick 
(10  times  the  protection  of  zinc  plating  -  3  microns) 

...the  “Conversation"  continues. ..What  Alternatives  might  satisfy  the  S&T  Exit  Criteria?  ....and  Why? 
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Worksheet  3.2 
Alternatives 
Complete  Descriptions 

(One  Worksheet  per  Alternative) 

EXAMPLE 


Project  Name:  AGE  Corrosion  Mitigation 

Member  Name: _ lack  Sparrow _ 

Role:  Team  Lead/Program  Manager 

Worksheet  Date: _ Today _ 


Describe  the  " attributes "  of  each  Alternative  in  as  great  a  detail  as  possible  as  they  pertain  to 
each  S&T  Exit  Criterion.  Some  form  of Description/Quad  Chart  presentation  (attached  to 
Worksheet  3.2),  could  make  value  assessment  and  decision  briefing  easier  as  you  go. 

Alternative  Name: _ Hot-Dip  Galvanization  ( HDG ) 

Description:  Process  whereby  fabricated  steel,  structural  steel,  castings,  or  small  parts. 
including  fasteners,  are  immersed  in  a  kettle  or  vat  of  molten  zinc,  resulting  in  a 

metallurgicallv  bonded  alloy  coating  that  protects  the  steel  from  corrosion. 

Any  Considerations:  Can  AGE  be  dis-assembled  into  smaller  components,  or  will  vat 
tank  need  to  be  large  enough  to  accommodate  large  assembled  units? 


Estimate  Cost  and  Schedule: _ 

Estimate  Initial  TRL: _ 6 _  Final  TRL: _ 9 

Estimate  Initial  MRL:  8  Final  MRL: _ 8 


Estimate  Payoff  to  Maturity  score:  (Score  the  Alternative  on  the  table  below) 


-j 

tc 

H 


. ]  Mature  ^  Mature  / 

;  Low  Pay°ff  k'A  Medium  Payoff 


Near  Term 
New  Capability 

Immature 
:  Game-Changer 


Payoff 


Dipping  AGE  into  molten  zinc 
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Step  4.  Evaluate  Alternatives  (Value  Analysis]  The  Measures  of  Merit 
are  Desirability  (Worksheet  4.1 )  and  Risk  (Worksheet  4.2] 


Worksheet  4.1 
Desirability  for  each 
S&T  Exit  Criteria 
Vs.  each  Alternative 

(One  Worksheet  per  Alternative) 

EXAMPLE 


Project  Name:  AGE  Corrosion  Mitigation 

Member  Name: _ lack  Sparrow _ 

Role:  Team  Lead/Program  Manager 

Worksheet  Date: _ Today _ 


Certain  Exit  Criteria  (EC)  are  more  important  than  other  Exit  Criteria.  Enter  an  EC  Weight 
value  from  Step  2  to  score  the  relative  importance  of  each  exit  criteria  compared  to  the  other 
exit  criteria. 

Certain  Alternatives  are  more  credible  to  attain  the  Exit  Criteria.  Enter  an  Alternative  (Alt) 
Score  as  to  how  well  the  Alternative  satisfies  each  S&T  Exit  Criterion  based  on  a  team  establish 
scoring  method.  If  any  Alternative  fails  to  meet  one  or  more  S&T  Exit  Criteria,  a  zero  score  is 
entered.  A  zero  score  essentially  fails  that  alternative  (EC  Wt  Score  x  0  -  0),  however,  it  is  still 
recommended  the  Alternative  be  retained  for  the  final  discussion. 

Multiply  EC  Weight  times  the  Alt  Score  (Wt  x  Sc)  for  an  "EC"  Score.  Add  all  the  EC  scores 
together  for  an  overall  Alternative  Score  (EC1  Score  +  EC2  Score  +  EC3  Score...) 


Alternate  Name:  Hot  Dip  Galvanization  (HDG) 

EC 

ID 

Exit 

Criteria 

EC 

Wt 

Alt 

Score 

EC  Score 

(WtxSc) 

Rationale 

P-1 

Durability 

1.0 

8 

8 

Thick  Zinc  Coating 

P-2 

EPA  /  ESH  Compliance 

0.7 

7 

5 

No  spray,  but  Zn  vapor 

P-3 

Surface  Coverage 

1.0 

9 

9 

Liquid  immersion 

P-4 

Repairability 

0.7 

8 

6 

Can  electroplate  repairs 

C-l 

Affordability 

1.0 

7 

7 

Equiv  to  current  method 

Alternative  Score  (EC1  Score  +  EC2  Score  +  EC3...) 

39 

35 

Note:  "3”  denotes  alternative  equivalent  to  the  capability  as  the  current  method, 

"7”  denotes  superior  capability  (meets  threshold  value),  and 
"10”  denotes  superior  capability  (meets  objective  value) 

DOCUMENT :  The  Team  should  negotiate  one  overall  Alternative  Desirability  score  for  each 
Alternative  that  is  then  transferred  and  compiled  on  the  Composite  Worksheet  4.3 


The  SE  " Conversation "  continues  "...Desirability  is  a  S&T  Exit  Criteria  value  with  room  for  flexibility" 
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Worksheet  4.2 
Risk  of  Achieving 
each  S&T  Exit  Criterion 
Vs.  each  Alternative 

(One  Worksheet  per  Alternative) 

EXAMPLE 


Project  Name:  AGE  Corrosion  Mitigation 

Member  Name: _ lack  Sparrow _ 

Role:  Team  Lead/Program  Manager 

Worksheet  Date: _ Today _ 


The  Team  identifies  the  top  3-5  Risks  for  each  Alternative  to  meet 
the  threshold  values  for  all  of  the  S&T  Exit  Criteria. 

The  Team  evaluates  the  Risk  of  each  Alternative  against  each  S&T  Exit  Criteria 
and  generates  a  Risk  Score  ( Risk  in  this  context  is  the  probability  of  failure  (Pf) 
to  exceed  the  Exit  Criterion  minimum  limits  (Worksheet  2.5)). 

Risk  does  not  mean  the  team  should  not  explore  all  possible  Alternatives 
( see  Payoff  to  Maturity  space  on  Worksheet  3.2 ).  High  Risk  simply  means 
the  team  should  consider  Risk  vs.  Payoff  plus  Mitigation!  The  score 
should  also  include  consideration  of  mitigation  difficulty. 


Alternate  Name:  Hot  Dip  Galvanization  (HDG) 

EC 

ID 

Exit 

Criteria 

Risk 

Description 

Risk 

Score 

Potential  /  Mitigation 

P-1 

Durability 

None 

10 

Highly  durable 

P-2 

EPA  /  ESH  Compliance 

Molten  metal 

7 

Use  remote  handling  to 
prevent  burns;  vapor 
recovery  system  needs 
to  be  installed 

P-3 

Surface  Coverage 

None 

10 

100%  coverage 
(open  pipes) 

P-4 

Repairability 

Field-level 

repairability 

9 

Apply  portable 
electroplating  brushes 
to  repair  small  areas 

C-l 

Affordability 

Size  of  hot-dip 
tanks 

8 

Assume  AGE  stands  can 

be  disassembled  into 
smaller  components 

Overall  Risk  Score:  _ 9  -  Low _ 

DOCUMENT:  The  Team  should  negotiate  one  overall  Risk  score  for  the  Alternative.  Each 
Alternative’s  overall  Risk  Score  is  then  transferred  to  the  Composite  Scorecard  -- 
Worksheet  4.3 


...the  "Conversation”  continues. ..What's  the  Probability  of  Failure  (Risk)  to  achieve  each  S&T  Exit  Criterion? 
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Worksheet  4.3 
Composite  Scorecard 

EXAMPLE  - 


Project  Name:  AGE  Corrosion  Mitigation 

Member  Name: _ lack  Sparrow _ 

Role:  Team  Lead/Program  Manager 

Worksheet  Date: _ Today _ 


Transfer  all  Alternative  and  Risk  scores  from  each  Alternative  Worksheets  (4.1  and  4.2 j 

to  this  Composite  Scorecard 


ALTERNATIVES 

EC 

ID 

Exit  Criteria 

Coatings  to 
MIL-PRF-26915 

Metallization 

Electro 

Galvanization 

Hot-Dip 

Galvanization 

P-1 

Durability 

3 

7 

5 

8 

P-2 

EPA  /  ESH  Compliance 

2 

2 

5 

5 

P-3 

Surface  Coverage 

2 

0 

6 

9 

P-4 

Repairability 

2 

4 

6 

6 

C-l 

Affordability 

3 

2 

8 

7 

Alternative  Score 

Single  Entry  per 
(Worksheet  4.1] 

12 

0 

29 

35 

Risk 

Single  Entry  per 
(Worksheet  4.2] 

10-  Low 

5  -  Medium 

10  -  Low 

9  -  Low 

Preferred  Alternative(s)  / 
Rationale 

Hot-Dip  Galvanization  is  the  best  solution.  Superior  capabilities  in 
durability  &  surface  coverage  offset  slightly  higher  cost  relative  to 
Electro-Galvanization;  risk  acceptable. 

Preferred  Alternative(s) 
Decision  Documentation 

The  Composite  Scorecard  is  not  the  final  answer,  but  presents  the  team  with  values  to  discuss  and 
agree  on  for  the  Preferred  Alternative^]. 

Analyze  the  scores  by  asking  the  questions: 

1  -  Does  the  ranking  of  the  composite  scores  pass  the  sanity  test,  i.e.,  do  the  values  make  sense? 

2  -  Do  Risks  outweigh  the  scores  for  the  high  scoring  Alternatives? 

Based  on  this  review,  the  team  moves  on  to  the  Action  Plan  for  the  Preferred  Alternative (s).  Often 
in  S&T  it  is  common  to  have  multiple  alternatives  selected  for  initial  development. 


...the  “Conversation”  continues. ..If  a  score  doesn't  make  sense,  ...discuss  why  it  scored  as  it  did? 
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Step  5.  Build  the  Plan:  With  the  Problem  Space  understood  and  the  Solution  Space  fully 
explored,  Advocacy  Briefs  and  Action  Plans  can  be  generation  on  the  Preferred  Alternative(s). 

Project  Name:  AGE  Corrosion  Mitigation 

Member  Name: _ lack  Sparrow _ 

Role:  Team  Lead/Program  Manager 

Worksheet  Date: _ Today _ 

The  Team  should  agree  upon  the  plan  to  advocate  for  approval/implementation  of  the 
proposed  S&T  program  incorporating  the  Preferred  Alternative (s).  Depending  on  the  nature 
of  the  proposed  program,  the  Action  Plan  could  range  from  a  simple  White  Paper  or  Roadmap 
to  a  fully  detailed  advocacy  briefing/package.  Worksheet  5.1  provides  a  possible  outline 
based  on  principles  of  good  program  management. 

Details  necessary  to  complete  this  Action  Plan  include  programmatic  technology 
performance,  costand  schedule  estimates,  a  basic  program  execution  strategy  of  external 
contracts  and  in-house  activities,  as  well  as  the  type  of  funding  (6.1,  6.2,  6.3)  required. 


Worksheet  5.1 
Program  Action  Plan 


Program  Action  Plan  Checklist 

□  Document  details  of  the  planning  process: 

□  AF  problem:  Corrosion  mitigation  costs  too  much  -  strains  sustainment  dollars 

□  Customer (s)/owners/end  users:  ALCs  and  ALL  Flight  Line  Users 

□  Requirements:  Costs,  Environment,  Performance 

□  S&T  exit  criteria:  Less  cost  than  current,  green,  highly  durable 

□  Details  of  each  alternative  solution  considered:  Four  options 

□  Preferred  alternative (s)  selected  (include  rationale):  Hot  Dip  Galvanization 

□  Document  proposed  program  execution/management  plan 

□  Proposed  S&T  program  schedule  (with  milestones  and  decision  points)  -  may  be 
in  the  form  of  a  technology  roadmap 

□  Required  resources 

□  Proposed  execution  approach  (inhouse,  external  contracts) 

□  Risk  management  approach 

•  Describe  the  recommended  next  steps  in  advocacy  process: 


...the  SE  “Conversation"  never  stops.. .A  well  thought  out  plan  is  easier  to  defend  and  execute. 
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APPENDIX  SECTION 


AFRL/RX 
STREAMLINED 
S&T  PLANNING 
PROCESS 


Tailored  Systems  Engineering  (SE)  Principles 
which  can  be  applied  to  plan 
Science  &  Technology  (S&T)  programs 


(Additional  reading  on  the  SE  Streamlined  Process) 
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APPENDIX  A  -  Acronyms 


AF 

Air  Force 

AFI 

Air  Force  Instruction 

AFMC 

Air  Force  Material  Command 

AFRL 

Air  Force  Research  Laboratory 

ATD 

Advanced  Technology  Demonstration 

CONOPS 

Concept  of  Operations 

COP 

Community  of  Practice 

COTS 

Commercial-Off-the-Shelf 

DAPA 

Defense  Acquisition  Performance  Assessment 

DAU 

Defense  Acquisition  University 

DoD 

Department  of  Defense 

EC 

Exit  Criteria 

EMD 

Engineering  Manufacturing  Development 

FOS 

Family  of  Systems 

GOTChA 

Goals,  Objectives,  Technical  Challenges,  and  Approach 

HVP 

High  Visibility  Program 

IPPD 

Integrated  Product  and  Process  Development 

IPT 

Integrated  Product  Team 

KPP 

Key  Performance  Parameter 

LMR 

Laboratory  Management  Review 

LRIP 

Low  Rate  Initial  Production 

MAJCOM 

Major  Command 

MRA 

Manufacturing  Readiness  Assessment 

MRL 

Manufacturing  Readiness  Level 

MS 

Milestone 

M&S 

Modeling  and  Simulation 

MS&A 

Modeling,  Simulation  and  Analysis 

PMR/PBR 

Program  Management/Program  Baseline  Review 

PM 

Program  Manager 

SE 

Systems  Engineering 

SEADS 

Systems  Engineering  Analysis  Decision  Support 

SEC 

Systems  Engineering  Council 

SETFST 

Systems  Engineering  Tailored  for  Science  and  Technology 

SEWG 

Systems  Engineering  Working  Group 

S&T 

Science  and  technology 

SME 

Subject  Matter  Experts 

SOS 

System  of  Systems 

SOTA 

State-of-the-Art 

TD 

Technology  Directorate 

TRA 

Technology  Readiness  Assessment 

TRL 

Technology  Readiness  Level 

V&V 

Verification  and  Validation 

WBS 

Work  Breakdown  Structure 

WS 

Worksheet 
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APPENDIX  C  -  Background  -  Systems  Engineering  (SE) 


What  is  Systems  Engineering? 

Systems  Engineering  (SE)  is  a  methodology  and  a  process  that  can  enhance  the 
likelihood  of  customer  and  stakeholder  needs  being  satisfied  in  a  high  quality,  trustworthy, 
cost  efficient,  and  schedule  compliant  manner  throughout  a  system's  entire  life  cycle, 
including  development. 

AFI  63-1201,  Life  Cycle  Systems  Engineering,  states  "Air  Force  SE  involves 
comprehensive  planning,  management,  and  execution  of  rigorous  technical  efforts 
to  develop,  field,  &  sustain  robust  products  and  systems." 

SE  collects,  coordinates,  &  ensures  traceability  of  all  stakeholder  needs  into  a  set  of 
system  requirements  through  a  balanced  process  that  takes  into  account  effectiveness, 
performance,  cost,  schedule,  and  risk. 

From  a  DoD  viewpoint,  SE  activities  are  based  around  the  following  eight  technical 
management  processes:  Technical  Planning,  Requirements  Management,  Interface 
Management,  Technical  Planning,  Configuration  Management,  Technical  Data  Management, 
Technical  Assessment,  and  Decision  Analysis. 


AFMCI63-1201,  14  October  2009 

Systems  Engineering  (SE):  SE  encompasses  the  entire  set  of  scientific,  technical, 
and  managerial  efforts  needed  to  conceive,  evolve  requirements,  develop,  verily 
capabilities,  deploy,  support,  sustain,  and  dispose  of  a  robust  product,  platform,  system, 
or  integrated  System-of-Systems/Family-of-Systems  (SoS/FoS)  capability  to  meet  user 
needs.  SE  may  be  referred  to  as  a  discipline,  a  methodology,  an  approach,  a  practice,  a 
process,  a  set  of  processes  and  sub-processes,  or  various  other  terms;  however,  its 
fundamental  elements  -  systematic  technical  and  managerial  processes  and 
measurements  -  remain  the  same  regardless  of  the  collective  nomenclature.  SE 
provides  the  integrating  technical  and  managerial  process  to  define  and  balance 
performance,  cost,  schedule,  risk,  supportability,  and  security  for  an  item,  system,  and 
SoS/FoS  throughout  their  life  cycle.  SE  requires  an  interdisciplinary  execution 
approach. 
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The  Air  Force  Mandate  for  Systems  Engineering 


The  2006  Defense  Acquisition  Performance  Assessment  (DAP A]  Project  Report  survey 
found  that  96%  of  respondents  cited  at  least  one  of  the  following  three  areas  as  critical  to 
maintaining  program  cost,  schedule,  and  performance  (shown  in  ranked  order): 

•  Requirements  instability 

•  Funding  instability 

•  Technology  maturity 

As  a  validated  method  to  improve  acquisition  program  management,  the  Air  Force  has 
formally  mandated  SE  in  the  form  of  two  instructions: 

•  AFI  63-1201,  Life  Cycle  Systems  Engineering ,  which  clarifies  and  emphasizes 
the  use  of  disciplined,  seamless  SE  practices  throughout  the 
concept/product/system  life  cycle 

•  AFI  63-101:  Acquisition  &  Sustainment  Life  Cycle  Management,  which 
designates  life  cycle  SE  as  one  of  the  six  tenets  of  Integrated  Life  Cycle 
Management  (ILCM)  and  emphasizes  SE  as  the  integrating  mechanism  for 
balanced  solutions 

AFI  63-1201  states  that  application  of  SE  fundamentals  must  begin  with  concept 
inception,  and  must  cover  all  efforts  across  all  life  cycle  phases,  to  include  sustainment  & 
disposal,  for  all  Air  Force  products  and  systems.  This  instruction  introduces  the 
terminology,  Early  SE,  which  provides  an  audit  trail  from  the  users’  capability  gaps  & 
needs,  through  concept  selection,  high-level  system  requirements  refinement,  & 
documentation  of  development  plans.  AFI  63-101  specifically  mentions  the  role  of  AFRL  in 
the  SE  process,  stating  "AFRL/CC  will  ensure  incorporation  of  SE  methodologies  tailored 
for  AFRL  technology  development  done  in  support  of  evolutionary  acquisition 
programs.” 
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The  Role  of  AFRL  in  Early  Systems  Engineering 


The  Air  Force  has  emphasized  the  benefits  of  Early  SE  to  the  extent  that  in  March 
2009  an  Air  Force  Early  Systems  Engineering  Guidebook  was  published.  This  document 
clearly  lays  out  the  role  of  AFRL  in  Early  SE,  stating  the  following: 

A  technology  organization,  typically  AFRL,  works  with  acquisition  organizations  to  ensure: 

•  Relevant  technologies  are  considered,  and  that  they  are  compatible  with  the 
desired  time  frame  and  expressed  acceptable  risk  levels 

•  New  approaches  made  possible  by  emerging  technologies,  as  well  as 
technologies  that  will  improve  a  system’s  effectiveness  and/or  reduce  its  cost 

•  Risks  and  uncertainties  associated  with  new  technologies  are  estimated  and 
impacts  are  assessed 

•  Insight  as  to  user/operator  needs  is  gained,  allowing  technologists  to  better 
focus  their  technology  roadmaps 

These  AFRL  roles  apply  to  both  evolutionary  acquisitions  for  fielded  Air  Force  systems  as 
well  as  to  newly  emerging  systems  such  as  the  long  range  strike  bomber.  Application  of 
tailored  SE  principles  will  greatly  assist  AFRL  scientists  and  engineers  as  they  engage  in  all 
levels  of  science  and  technology. 
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Advantages  of  Applying  Early  SE  in  Science  &  Technology  (S&T) 


What  is  the  benefit  of  applying  early  or  tailored  SE  in  S&T  programs?  The  2006  Defense 
Acquisition  Performance  Assessment  (DAP A )  Project  Report  states  that  the  greatest  trade  space,  and 
thus  the  largest  risk  reduction  opportunity  in  the  DoD  Acquisition  Life  Cycle,  exists  between 
Milestones  (MS]  A  and  B  (Figure  1],  A  second  major  finding  was  that  for  many  major  DoD 
acquisition  programs,  balancing  and  integration  of  technology  maturity,  system  capability,  cost  and 
program  risk  is  not  being  achieved  and  agreed  to  prior  to  Milestone  B,  thereby  engendering 
excessive  cost,  schedule  and  performance  risk. 

It  is  sometimes  difficult  to  justify  Early  SE  in  terms  that  AFRL  program  managers  and  their 
supervisory  chain  can  relate  to.  The  costs  of  Early  SE  are  immediately  evident  in  terms  of  resources 
(people  &  funding]  and  schedule  time.  The  benefits,  although  sometimes  less  obvious  in  the  near 
term,  have  been  shown  in  several  studies  to  significantly  improve  the  quality  of  S&T  deliverables 
over  the  long  haul.  These  benefits  include: 

•  Cost  avoidance  (reduction  of  rework  from  requirements  shift  or  interface 
mismatches] 

•  Risk  management  (early  risk  identification  and  mitigation] 

•  Improved  efficiency  (clearer  organizational  boundaries  and  interfaces] 

•  Better  technology  products  (resulting  from  a  better  understanding  and  satisfaction 
of  customer  needs] 


Early  SE  Focus  (Oooi 5000.02) 


Figure  11.  Greatest  Trade  Space/Largest  Risk  Reduction  Opportunity  -  Exists  Between 

Milestones  (MS]  A  &  B 
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AFRL  lies  at  the  heart  of  this  maximum  payoff  trade  space;  and  researchers,  engineers,  and 
designers  who  have  employed  tailored  SE  consistently  report  that: 

•  Teams  gain  clarity  on  customer  requirements,  making  it  much  easier  to  define 
technology  and  program  goals  early  on 

•  Previously  overlooked  technical  and  programmatic  issues  are  now  routinely  considered 
up  front 

•  Applied  research  (6.2]  and  advanced  development  [6.3]  programs  focus  more  on 
customers,  and  consciously  think  through  potential  transition  issues 

•  Program  advocacy  is  much  stronger  and  more  effective 

•  Programs  gain  continuity  and  stay  the  course  in  spite  of  personnel  turnover 

•  The  process  enhances  funding  continuity  and  opens  new  avenues  for  external  funding 

•  The  process  provides  an  enhanced  ability  to  revisit  established  courses  of  action 

When  tailored  SE  is  applied  by  S&T  teams,  a  number  of  difficulties  that  persist  in  attempts  to 
successfully  execute  S&T  programs  are  reduced  or  alleviated,  including: 

•  Requirements  creep 

•  Ineffective  customer  engagement  or  buy-in 

•  Insufficient  technology  maturity  planning 

•  Unproductive  technology  transition  planning 

•  Weak  sponsor  or  user  commitment 

•  Poor  basis  for  decisions  driven  by  resource  constraints 


New  AFRL  Environment:  competition  for  resources, 
emphasis  on  integration,  teamwork  is  essential. 
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History  of  SE  in  AFRL 

For  more  than  a  decade,  AFRL  has  sustained  a  commitment  to  enhance  their  core  business 
processes  through  the  tailored  application  of  SE  in  both  the  planning  and  the  execution  of  the  S&T 
portfolio. 

Leading  the  other  military  services  in  compliance  with  direction  from  an  OSD  Affordability 
Task  Force,  AFRL  tasked  its  Technology  Directorates  (TDs]  in  February  2000  to  implement  an 
Affordability  Policy  during  the  execution  of  advanced  technology  development  programs.  This 
initiative  was  based  on  a  strategy  of  tailoring  SE  methods  that  were  proven  successful  in  industry 
for  use  in  the  Air  Force  S&T  environment. 

In  July  2002,  AFRL  established  an  Acquisition  Center  of  Excellence  [AFRL/AE]  to  lead  a 
transformation  in  the  way  the  latest  technologies  were  planned,  developed,  and  delivered  to  the 
AFRL  customer  base.  A  memorandum  signed  in  February  2003  by  AFRL/CC  and  SAF/AQR  detailing 
a  new  Technology  Transition  Initiative  was  followed  by  a  memorandum  signed  in  July  2003 
directing  implementation  of  a  series  of  actions.  Among  these  actions  were  imperatives  to  ensure 
implementation  of  SE  principles  among  the  AFRL  TDs  and  to  craft  a  comprehensive  SE  strategy  for 
AFRL  that  would  complement  SE  initiatives  then  under  development  by  other  acquisition 
organizations  in  the  Air  Force. 

In  2005,  the  AFRL  Systems  Engineering  Working  Group  (SEWGJ  achieved  Command  Section 
signature  and  release  of  a  new  AFRL  Instruction,  AFRLI  61-104  "Science  and  Technology  SE 
Initiative."  The  AFRLI  61-104  Attachment  2,  "Eight  (8]-Key  SE  Questions"  provides  a  foundation  for 
AFRL/RX  SE  Assessment  Standards  in  the  conduct  of  Program  Baseline  Reviews  (PBRJ,  Laboratory 
Management  Reviews  (LMRJ,  and  Technical  Management  Reviews. 

The  same  8-Key  SE  Questions  are  asked  across  the  entire  AFRL  S&T  portfolio.  While  every 
S&T  program  manager  is  expected  to  know  the  answers  to  these  questions,  the  amount  of 
knowledge  needed  to  answer  a  particular  question  satisfactorily  (i.e.  at  LMRs]  will  change  as  a 
program  matures  from  6.1  to  6.2,  and  eventually  to  6.3. 

How  do  we  do  it?  AFRL  "8-Key  Questions"  per  AFRLI  61-104,  Attachment  2 

1.  Who  is  your  customer?  ...and  Why? 

2.  What  are  the  Customer's  requirements?  ...and  Why? 

3.  How  will  you  demonstrate  you  have  met  the  requirements?  ...and  Why? 

4.  What  are  the  technology  options?  ...and  Why? 

5.  Which  is  the  best  approach?  ...and  Why? 

6.  What  are  the  risks  to  developing  the  selected  technology?  ...and  Why? 

7.  How  will  you  structure  your  program  to  meet  requirements  and  mitigate  risk?  ...and 

Why? 

8.  What  is  your  business-based  transition  plan  that  meets  customer  approval?  ...and  Why? 
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AFRLI  61-104  "Eight  Key  Questions" 

Figure  12.  8-  Key  Questions  presented  as  Systems  Engineering  'Vee' 


By  2007,  a  senior-level  Systems  Engineering  Council  (SEC)  had  been  established  in 
AFRL,  chaired  by  the  AFRL/XP  Deputy  for  SE  and  Program  Management,  and  including  a 
Chief  Engineer  or  senior  engineer  from  each  AFRL  TD.  The  primary  objectives  of  the  SEC 
are  to  institutionalize  the  adoption  of  SE  across  AFRL  and  to  leverage  SE  best  practices.  A 
principal  focus  of  this  council  is,  as  it  has  been  from  the  outset,  to  appropriately  tailor  SE 
methods  for  use  across  the  S&T  community. 

As  mentioned  earlier,  a  variety  of  SE-related  formal  guidance  has  been  provided 
from  the  DOD  and  the  Air  Force  itself.  Appendix  G  gives  a  more  detailed  sampling  of  this 
guidance. 
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APPENDIX  D  -  Conducting  Brainstorming  Sessions 


Brainstorming:  a  group  creativity  technique  designed  to  generate  a  large  number  of  ideas 
to  solve  a  problem.  Brainstorming  is  a  special  form  of  problem  solving,  where  the  mind  seeks  to 
see  solutions  never  before  thought. 

individual  Brainstorming  before  and  after  Group  Sessions:  Creativity 
comes  from  a  blend  of  both  the  individual  and  the  collective  "ideation."  This  means  providing 
time  for  people  to  think  and  learn  about  the  topic  before  the  group  brainstorm,  as  well  as  time  to 
reflect  about  what  happened  after  the  meetings. 

Encourage  the  right  mindset  and  have  fun.  Consider  using  an  ice-breaker  or  creativity 
exercise  to  get  group  members  into  the  right  frame  of  mind.  Keep  the  exercise  fun. 

Ground  Rules: 

Stated  in  the  memo  and  at  the  beginning  of  the  session,  there  are  four  basic  rules  in 
brainstorming  intended  to  reduce  social  inhibitions  among  group  members,  stimulate  idea 
generation,  and  increase  overall  creativity  of  the  group. 

1.  Quantity:  Facilitate  problem  solving  through  the  maxim  quantity  breeds  quality. 
Generate  as  many  ideas  as  possible  in  a  certain  amount  of  time.  The  assumption  is 
the  greater  the  number  of  ideas  generated,  the  greater  the  chance  of  producing  a 
radical  and  effective  solution 

2.  Withhold  judgment  and  criticism:  Focus  on  extending  or  adding  to  ideas, 
reserving  criticism  for  a  later  'critical  stage'  of  the  process.  By  suspending 
judgment,  participants  will  feel  free  to  generate  unusual  ideas 

3.  No  Idea  too  Stupid  —  Encourage  unusual  ideas:  To  get  a  long  list  of  good  ideas, 
the  ‘stormers’  must  look  from  new  perspectives  and  be  creative 

4.  Combine  and  improve  ideas:  Good  ideas  can  be  combined  to  form  a  single  better 
good  idea. 
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Set  the  problem 


Before  a  brainstorming  session,  it  is  important  to  define  the  problem  for  which  the 
altemative/solutions  will  be  generated.  The  problem  must  be  clear,  not  too  big,  and  captured  in  a 
specific  question.  If  the  problem  is  too  big,  the  facilitator  should  break  it  down  into  smaller 
components,  each  with  its  own  question. 

Create  a  list  of  lead  questions 

During  the  brainstorm  session  the  creativity  may  decrease.  At  this  moment,  the 
facilitator  should  stimulate  creativity  by  suggesting  a  lead  question  to  answer,  such  as  Can  we 
combine  these  ideas  or  look  at  from  another  perspective? 

Conduct  Session 

The  facilitator  leads  the  brainstorming  session  and  ensures  the  ground  rules  are  followed. 
Possible  steps  in  a  typical  session  are: 

1.  A  warm-up  session,  to  expose  novice  participants  to  the  criticism-free  environment. 
A  simple  problem  is  brainstormed,  for  example  What  should  be  the  CEO's  retirement 
present?  or  What  can  be  improved  in  Microsoft  Windows?. 

2.  The  facilitator  presents  the  problem  and  gives  a  further  explanation  if  needed. 

3.  The  facilitator  asks  the  group  for  their  ideas. 

4.  If  no  ideas  are  forthcoming,  the  facilitator  suggests  a  lead  to  encourage  creativity. 

5.  All  participants  present  their  ideas,  and  the  idea  collector  records  them. 

6.  To  ensure  clarity,  participants  may  elaborate  on  their  ideas. 

7.  When  time  is  up,  the  facilitator  organizes  the  ideas  based  on  the  topic  goal  and 
encourages  discussion. 

8.  Ideas  are  categorized. 

9.  The  whole  list  is  reviewed  to  ensure  everyone  understands  the  ideas. 

10.  Duplicate  ideas  and  obviously  infeasible  solutions  are  removed. 

11.  The  facilitator  thanks  all  for  participating. 
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The  Process 


•  Participants  who  have  ideas  but  were  unable  to  present  them  are  encouraged  to 
write  their  ideas  down  and  present  them  later. 

•  The  idea  collector  should  repeat  the  idea  in  the  words  he  or  she  has  written 
verbatim,  to  confirm  that  it  expresses  the  meaning  intended  by  the  originator. 

•  When  many  participants  are  having  ideas,  the  one  with  the  most  associated  idea 
should  have  priority.  This  is  to  encourage  elaboration  on  previous  ideas. 

•  During  a  brainstorming  session,  managers  and  other  superiors  may  be  discouraged 
from  attending,  since  it  may  inhibit  and  reduce  the  effect  of  the  four  basic  rules, 
especially  the  generation  of  unusual  ideas. 

Evaluation 

Usually  the  group  itself  will  evaluate  the  ideas  and  select  one  or  possibly  two  approaches  as 
potential  solution(s)  to  the  problem. 

•  The  solution  should  not  require  resources  or  skills  the  members  of  the  group  do  not 
have  or  cannot  acquire. 

•  If  acquiring  additional  resources  or  skills  is  necessary,  that  needs  to  be  the  first  part 
of  the  solution. 

•  There  must  be  a  way  to  measure  progress  and  success. 

•  The  steps  to  carry  out  the  solution  must  be  clear  to  all,  and  amenable  to  being 
assigned  to  the  members  so  that  each  will  have  an  important  role. 

•  There  must  be  a  common  decision  making  process  to  enable  a  coordinated  effort  to 
proceed,  and  to  reassign  tasks  as  the  project  unfolds. 

•  There  should  be  evaluations  at  milestones  to  decide  whether  the  group  is  on  track 
toward  a  final  solution. 

•  There  should  be  incentives  to  participation  so  that  participants  maintain  their 
efforts. 
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APPENDIX  E  -  Work  Breakdown  Structure  (WBS) 


A  WBS  is  one  approach  to  help  structure  the  problem  and  to  establish  the 
relationships  of  systems  necessary  to  address  all  functional  considerations. 


WBS:  Define  the  work  (elements)  of  the  project...  a  "Progressive  Elaboration" 
deliverables-oriented  hierarchy. 


Level  1:  Major  System  (per  MILSTD  881  A:  Aircraft,  Missile,  Ordnance,  Space,  Sea,  Surface  Vehicle,  etc.] 

Level  2:  Major  elements  of  Level  1  (Air  Vehicle,  Cost,  Test  &  Evaluation,  Training,  Systems  Engineering,  etc.] 
Level  3:  Subdivided  Level  2  elements  (Frame,  Power,  Displays  &  Control,  Com/Identification,  etc.] 


Level  1 


Aircraft  System 


Level  2 


Air 

Vehicle 


SE  / 

Program 

Mgmt 


System 

T&E 


Training 


Aircraft  Systems  WBS 
(MIL-HDBK-881) 


Data 


Peculiar 

Support 

Equipment 


Common 

Support 

Equipment 


Op/Site 

Activation 


Industrial 

Facilities 


Airframe 
Propulsion 
Application  Software 
System  Software 
Com/Identification 
Navigation/Guidance 
Central  Computer 
Fire  Control 

Data  Display  and  Controls 
Survivability 
Reconnaissance 
Automatic  Flight  Control 
Central  Integrated  Checkout 
Antisubmarine  Warfare 


DT&E 

OT&E 

Mockups 

T&E 

Support 

Test 

Facilities 


Equipment  Tech  Pubs  Test  and  Test  and 


Services 

Facilities 


Engrg  Data 

Support 

Data 

Manage¬ 
ment  Data 
Data 

Depository 


Measurem’t  Measurem't 
Equipment  Equipment 

Support  Support 

and  and 

Handling  Handling 
Equipment  Equipment 


Sys 

Assembly, 

Installation 

and 

Checkout 
on  Site 

Contractor 
Tech  Support 

Site 

Construction 

Site/Ship 

Vehicle 

Conversion 


Initial 

Spares  and 
Initial 
Repair 
Parts 


Construc- 
tion/Conver- 
sion/Expan- 
sion 

Equipment 
Acquisition 
or  Mod 

Maintenance 


(Specify  by 
Allowance 
List, 

Grouping 

or  raw 

Element) 


Level  3 


Armament 
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Figure  13.  WBS  Example  to  functionally/systematically  breakdown 
elements  of  the  problem/environment 
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APPENDIX  F  -  Lessons  Learned  Regarding  SE  in  S&T 

The  following  aspects  are  often  essential  to  successful  planning  of  an  S&T  program  or  project: 

•  Developing  documentation 

•  Maintaining  the  team  throughout  the  project 

•  Sticking  to  the  plan 

•  Coordinating  with  the  full  team  as  things  change 

•  Non-participating  Scribe  role 

•  Building  plans  and  briefings  from  RX  Streamlined  SE  Core  Process  products 

•  The  "Team”  owns  each  step  in  the  process  and  its  products 

A  fixed  number  of  focused  team  meetings  should  be  defined  in  the  team  charter  to  scope  the  extent  of 
the  program  or  project  planning,  understanding  that  significant  changes  can  lead  to  iterative  planning  cycles. 

Lessons  Learned  from  AFRL/RX  Case  Studies: 

•  Process  provided  a  disciplined  analysis  of  options. 

•  Process  identified  need  to  interface  with  strong  customers  outside  AFRL  (even  to  DOD  and 
industry) 

•  Process  established  a  consistent  framework  for  information  exchange  and  synthesis  among 
managers  and  Subject  Matter  Experts  (SMEs). 

•  Challenges  to  process  effectiveness  include  dependence  on  team  participation  and  SME  support. 

•  Customer/User  involvement  in  the  process  is  extremely  valuable;  enabling  the  team  to  gain 
insights  that  otherwise  would  not  have  been  captured. 

•  Process  is  effective  as  an  analysis  and  a  decision-supporting  tool  in  that  it  reveals  sensitivities 
and  quantifies  risks. 

•  Process  is  easily  updated  as  knowledge  evolves. 

•  Execution  of  even  a  part  of  the  process  accelerates  discovery  and  causes  redirection  of  thought 
and  effort  into  more  fruitful  avenues. 

•  Team  members  learned  from  application  of  the  process  about  other  tools,  and  surfaced 
unspoken  issues. 

•  Structured  dialogue  driven  by  the  process  codified  thinking. 

•  Streamlining  the  process  and  motivating  participants  are  important. 

•  Need  to  establish  a  timeline  and  follow  it. 
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APPENDIX  G  -  Systems  Engineering  Guidance 


National  Research  Council 

Pre-Milestone  A  and  Early-Phase  Systems  Engineering,  2008  Study  Report 

•  Most  critical  SE  activities  during  Pre-Milestone  A/B  are 

o  Consistent  and  coordinated  user  requirements 
o  Clear  definition  of  Key  Performance  Parameters 
o  Analysis  of  Alternatives 

o  Structured  user,  acquirer,  industry,  sponsors,  and  S&T  collaboration 
o  CONOPS 

o  Assessment  of  system  performance  with  Modeling,  Simulation  and  Analysis  (MS&A) 
o  Architecture 
o  Risk  assessment 

o  Consensus  on  how  requirements  will  be  tested 
o  Technology  Maturation  Plan 
o  Establishing  cost  credibility 

Under  Secretary  of  Defense  (Acquisition  Technology  &  Logistics) 

Policy  for  Systems  Engineering  in  DoD,  20  February  2004  memo 

•  SE  must  be  embedded  in  planning  and  performed  across  acquisition  life  cycle 
Systems  Engineering  Plan  (SEP)  Preparation  Guide,  24  September  2004 

•  Documentation  will  include 

o  Requirements 
o  Tailored  SE  processes 
o  Entry  and  exit  criteria 

Policy  Addendum  for  Systems  Engineering,  22  October  2004 

•  Program  reviews  should  be  event  driven 

•  Peer  reviews  should  be  accomplished 

•  SE  best  practices  in  the  Defense  Acquisition  Guide  should  be  tailored  to  meet  program  needs 
Defense  Acquisition  Guidebook,  https:  //akss.dau.mil/dag/ 

•  SE  processes  are  applied  early  in  concept  definition,  and  then  continuously  throughout  the 
total  life  cycle 

•  Balanced  solutions  are  best  achieved  by  applying  SE  to  planning,  development,  and 
implementation  of  a  system 

•  Relevant  technologies  are  considered  that  they  are  compatible  with  the  desired  time  frame 
and  express  acceptable  risk  levels 

•  New  approaches  made  possible  by  emerging  technologies,  as  well  as  by  technologies  that 
will  improve  a  system’s  effectiveness  and/or  reduce  its  cost 

•  Risks  and  uncertainties  associated  with  new  technologies  are  estimated,  and  impacts  are 
assessed 

•  Insight  as  to  user/operator  needs  is  gained,  allowing  technologists  to  better  focus  their 
technology  roadmaps 
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SAF /AQ  Policy  on  Life  Cycle  Systems  Engineering — Concept  Phase 

Air  Force  Instruction  63-1201,  Life  Cycle  Systems  Engineering,  23  July  2007 

•  Fundamental  elements  of  SE  are  technical  processes  and  measurements 

•  SE  stakeholders  include  researchers,  acquirers,  developers,  users,  operators,  testers, 
trainers,  maintainers,  and  sustainers 

•  Application  of  SE  must  begin  with  concept  inception 

•  SE  involves  comprehensive  planning  and  addresses  architecting,  requirements  development 
and  management,  design,  technical  management,  test  and  evaluation,  and  verification  and 
validation 

•  Early  SE  provides  an  audit  trail  from  the  users’  capability  gaps  &  needs,  through  concept 
selection,  high-level  system  requirements  refinement,  &  documentation  of  development 
plans 

Technology  Transition  Initiative,  signed  4  February  2003  by  AFRL  and  SAF/AQ 

•  AFRL’s  portfolio  will. ..incorporate  systems  engineering  methods  tailored  to  the  nature  of 
specific  technology  programs 

•  SE  will  be  used  to  achieve  best  value  results,  accounting  for  the  critical  considerations  of 
customer  requirements,  exit  criteria,  technology  assessment,  risk  and  schedule 

Air  Force  Research  Laboratory 

AFRLI  61-104, 17  March  2008 

•  The  objectives  of  the  application  of  SE  to  AFRL  S&T  programs  are: 

o  Stronger  AFRL  program  management  and  decision-making  processes  and  competencies 

o  Alignment  and  integration  of  the  AFRL  technology  transition  process  with  those  used  by 
customer  organizations 

o  Consistent  delivery  of  technology  products  that  represent  best  value  solutions  to  needed 
warfighting  capabilities.  Best  value  is  defined  as  the  optimum  balance  of  technology 
solutions  that  meet  (both)  customer  requirements  and  the  transition  risk  associated 
with  successful  acquisition  of  those  technology  solutions 

o  Improved  rate  of  successful  technology  transition  to  the  customers 
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APPENDIX  H  -  TRL  vs.  MRL 


S&T  Exit  Criteria,  which  include  S&T  Key  Performance  Parameters  (KPPs),  are  levels 
of  measurable  performance  (thresholds  and  objectives)  that  must  be  achieved  for  program 
success.  The  demonstrated  performance  within  an  S&T  environment  is  used  to  establish  a 
given  technology  maturity  level,  known  as  a  Technology  Readiness  Level  (TRL)  (Figure  14). 

S&T  Exit  Criteria  are  particularly  important  for  S&T  programs  with  transitionable 
technology  products,  where  these  metrics  will  need  to  be  achieved  and  validated  in  a  way 
that  follow-on  acquisition  programs  have  acceptable  risk  in  meeting  the  requirement  set 
defined  in  Step  1. 

An  AFRL  6.3  ATD  program,  for  example,  will  typically  mature  a  given  technology 
deliverable  only  to  a  TRL  value  of  5  or  6  before  transitioning  to  an  acquirer  or  other 
customer.  In  the  case  of  basic  or  applied  research  programs  delivering  more  immature 
technologies  to  internal  AFRL  customers,  the  TRL  deliverable  level  may  be  at  a  much  lower 
value. 


Engineering  & 
Manufacturing 
Development 


Technology 

Demonstration 

Technology 

Development 


Research  to 
Prove  Feasibility 


Basic  f  | 
Research  v 


Actual  system  “Flight  Proven"  through  successful 
mission  operations 

Actual  system  completed  and  “flight  qualified” 
through  test  and  demonstration 

System  prototype  demonstration  in  an  operational 
environment 

System/subsystem  model  or  prototype 
demonstration  in  a  relevant  environment 

Component  and/or  breadboard  validation  in  a 
relevant  environment  -  environment  can  be  simulated 

Component  and/or  breadboard  validation  in  a 
laboratory  environment 

Analytical  &  experimental  critical  function  and/or 
characteristic  proof-of-concept 

Technology  concept  and/or  application  formulated 

Basic  principles  observed  and  reported 


Figure  14. 


Technology  Readiness  Levels  (TRL)  (Definitions  taken  from  DoD  5002) 
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TRLs  are  not  the  only  measure  of  the  maturity  of  technology.  It  is  very  possible  for  a 
technology  to  meet  all  the  definitions  of  a  TRL  6  yet  be  unacceptable  for  transition  to  the 
customer.  For  this  reason,  additional  dimension  of  maturity  should  be  evaluated  or 
considered  for  a  given  technology  solution.  Software  Readiness  Levels  (SRL)and 
Manufacturing  Readiness  Levels  (MRL)  are  two  common  areas  of  consideration.  Figure  15 
shows  the  MRL  definitions. 


Production  & 
Deployment 

MRL 

10 

MRL  9 

Engineering  & 
Manufacturing 
Development 

MRL  8 

MRL  7 

Technology 

Development 

MRL  6 

Material  Solutior 
Analysis 

= 

MRL  5 

MRL  4 

Pre-Concept 

Refinement 

Full  Rate  Production  demonstrated  and 
lean  production  practices  in  place 

Low  Rate  Production  demonstrated. 

Capability  in  place  to  begin  Full  Rate  Production 

Pilot  line  capability  demonstrated. 

Ready  to  begin  low  rate  production 

Capability  to  produce  systems,  subsystems  or 
components  in  a  production  representative  environment 

Capability  to  produce  a  prototype  system  or  subsystem 
in  a  production  relevant  environment 

Capability  to  produce  prototype  components 
in  a  production  relevant  environment 

Capability  to  produce  the  technology 
in  a  laboratory  environment 

Manufacturing  concepts  identified 


Figure  15.  Manufacturing  Readiness  Levels  (MRL)  Definitions 
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Workbook  Intro 


The  Worksheets  in  this  Workbook  are  to  foster  a  TEAM  approach  to  decomposing  a 
problem  and  to  using  the  principles  of  Systems  Engineering  to  stimulate  creativity  and 
discussion. 

Detailed  instructions  for  each  Worksheet  are  available  in  the  Streamlined  S&T  Planning 
Guide. 


Problem  Space 


Solution  Space 


Step  1 

FORM  TEAM 


Step  2 

REQUIREMENTS 


Step  3 

ALTERNATIVES 


Step  4 
EVALUATE 


Step  5 

DELIVER  PLAN 


The  above  Worksheet  activities  can  be  used  to  guide  your  Team’s  process  and 
are  meant  to  be  customized,  expanded,  or  changed  to  suit  your  purpose. 
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Step  1:  Form  Team.  Start  to  think  of  the  team  in  specific  terms  of  the  problem  at  hand. 
Different  problems  require  different  member  experiences  and  skills. 


Worksheet  1.1 
Define  Problem 


Project  Name 
Member  Name 
Role 

Worksheet  Date 


The  team  works  with  the  Customer (s) /Stakeholde r(s)  to  scope  the  problem  space. 
Who  is  the  Customer?  (Who  brought  the  problem?) _ 


Time  frame  of  Problem  (When  Needed/Urgency?): _ 

Describe  the  Problem  (Scope/Major  Issues/Constraints): 


Who  is  the  Problem  Owner?  (Who  has  the  task  to  solve  the  problem ?) 


Who  is  the  End  User?  (Who  turns  the  wrench/pushes  the  button?) 


Why  should  AFRL/RX  be  working  this  issue? 


Who  Needs  to  be  included  as  Partners  in  this  issue? 


What  Do  We  NOT  Know? 


...  the  "SE  Conversation"  begins  with  understanding  of  the  problem.. .What  is  the  "Discontent?”  ...and  Why? 


98 

Approved  for  public  release. 


Worksheet  1.2 
Team  Make-up  /  Roles 


Project  Name: _ 

Member  Name: _ 

Role: _ 

Worksheet  Date: _ 

With  the  Problem  understood,  a  team  approach  (Core  plus  Augmentees) 
must  by  defined,  i.e.,  identify  the  skills  needed  to  discuss  and  refine  the  Problem 

Team  Directory,  Roles  &  Responsibilities 

Role  /Responsibility  ORG/Svmbol  Name  /  Contact  Info 


Augmentees  (includes  SMEs,  Finance,  Contracting,  etc.) 


...the  "SE  Conversation"  continues...  including  all  the  right  expertise  and  Stakeholders 
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Project  Name: _ 

Worksheet  1.3  Member  Name: _ 

Establish  T earn  Charter  Role: - 

Worksheet  Date: _ 

The  Core  Team  stays  with  the  entire  process.  Other  persons  (Augmentees) 
are  brought  in  with  needed  expertise  where  appropriate. 

Use  Worksheets  1.1  and  1.2  as  backup  to  this  activity 

“Planning”  Team  Charter 

•  Goal/Objectives: _ 


•  Schedule 

o  Meeting  frequency  (Weekly,  monthly,  etc.) _ 

O  Duration:  Start:  (mm/dd/yyyy)  _  End:  ( mm/dd/yyyy )  _ 

•  Resources  ( Available  to  the  Team):  _ 

•  Authority  /  Accountability  of  Team  Membership:  _ 

As  a  Core  Team  Member  for  this  planning  effort,  I  understand 
the  following  is  expected  of  all  Core  Team  Members: 

o  Commit  to  participate  in  all  team  meetings 
o  Commit  to  complete  all  "homework"  on  schedule 

o  Participate  in  the  documentation  of  each  step  before  proceeding  to  the 
next  step 


Signatures  of  all  core  planning  team  members  commitment 


4. 

4. 

5. 

5. 

6. 

6. 

(The  above  is  just  a  suggested  outline.  Expand  with  extra  pages  if  needed) 


..  a  Team  Charter  formalizes  expectations,  removes  doubt,  and  improves  the  SE  “Conversation” 
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Step  2:  Requirements.  A  solid  understanding  of  the  requirements,  technical  challenges,  and 
how  they’re  stated  as  S&T  Exit  Criteria  are  critical  steps  in  Problem  Space  documentation. 


Worksheet  2.1 
List  of  Customer 
Requirements 


Project  Name: . 
Member  Name: . 

Role: . 
Worksheet  Date: 


The  Core  Team  refines  the  understanding  of  the  Customer’s  Problem 
into  stated  requirements  and  validates  them  with  the  Customer 


Requirement 

Name 

Requirement  Description 

(Be  as  specific  as  possible] 

Threshold* 

Objective* 

*Desirable  -  vary  with  maturity  of  task 


Threshold:  The  measurable  value  that  must  be  achieved  for  the  technology  to  advance  to  the  next 
stage  of  development  or  transition  ( usually  given  as  a  minimum  or  maximum  value). 

Objective:  The  stretch-goal  value  that  is  desirable,  but  not  essential  (often  viewed  as  the 
requirements  trade  space  if  parameters  such  as  cost  or  schedule  are  more  important  to  the  customer). 


□  DOCUMENT:  Validate  this  worksheet  with  your  customer  representatives /stakeholders 
to  ensure  agreement  with  the  goals. 


...the  “Conversation”  continues...  What  capability  does  my  Customer  (Want)  Need?  ...and  Why? 
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Project  Name: . 

Worksheet  2.2  Member  Name: . 

Technical  Challenges  Roie:_ 

Worksheet  Date: 


The  Team  now  has  to  turn  the  Customer  Requirements  into  actionable 
S&T  descriptions  (S&T  Exit  Criteria ).  This  step  helps  the  team  decompose 
the  Customer  Requirements  into  Technical  Issues  (Challenges). 


Requirement  Name 

Technical  Challenge  Issues 

....the  SE  Conversation  continues  by  identifying  "Where's  the  S&T  in  this  challenge?”. ..and  Why? 
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Worksheet  2.3 
List  of 

S&T  Exit  Criteria 


Project  Name: 
Member  Name: . 

Role: . 
Worksheet  Date: 


Customer  Requirements  are  usually  at  a  fairly  high  level.  The  team  now  has  to  define  and 
document  what  a  final  product,  technology,  or  system  must  do,  including  the  parameters 
that  define  successful  completion.  ( Worksheets  2.1  and  2.2)  should  lead  into  this  step. 


Criteria  Name 

Description 

Threshold* 

Objective* 

RQT  Name 

*Desirable  -  vary  with  maturity  of  task 

Threshold:  The  measurable  value  that  must  be  achieved  for  the  technology  to  advance  to  the  next 
stage  of  development  or  transition  (usually  given  as  a  minimum  or  maximum  value). 

Objective:  The  stretch-goal  value  that  is  desirable,  but  not  essential  (often  viewed  as  the 
requirements  trade  space  if  parameters  such  as  cost  or  schedule  are  more  important  to  the  customer). 


...the  “Conversation”  continues...  What  S&T  Exit  Criteria  demonstrates  the  Requirement(s)  have  been  met?"  - 

....and  Why? 
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Worksheet  2.4 
Group  S&T 
Exit  Criteria 
by  Category 


Project  Name: 
Member  Name: . 

Role: . 
Worksheet  Date: 


S&T  Exit  Criteria  negotiated  by  the  team  can  usually  be  grouped  into  common  topics, 
identified  such  as  P-1,  P-2,  etc.  for  Performance,  C-l,  C-2,  etc.  for  Cost. 


There  is  no  limit  to  the  number  of  S&T  Exit  Criterion  but  this  is  the  logical  place 
to  consider  consolidation,  combining  the  criteria,  if  appropriate. 


ID 

S&T  Exit 

Criteria 

Description 

Group: 

Group: 

Possible  Category  Groupings  (add  categories  depending  on  technology) 


C=  Cost  (development,  acquisition,  deploy,  point  of  use  operation) 
S  =  Schedule 

P  =  Performance  (reliability,  weight,  footprint,  set-up  time) 

L  =  Logistics  (transport,  service  life,  storage,  scalability,  disposal 


HF  =  Human  Factors 

E  =  Environment  (EPA,  certifications,  etc.) 

ST  =  Strategic  (outside  influences,  be  they  political  or  otherwise) 
POL  -  Political  (constrictions) 


...the  “Conversation”  continues...  What  groups  form  the  S&T  Exit  Criterion?"  -....and  Why? 
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Worksheet  2.5 
S&T  Exit  Criteria 
Complete  Description 


Project  Name: _ 

Member  Name:  _ 

Role: _ 

Worksheet  Date: 


The  Team  defines  the  detailed  quantitative  attributes  of  the  S&T  Exit  Criteria. 


ID 

Name 

Priority 

/Wt* 

Units 

Threshold 

Value 

Threshold 

Rationale 

Objective 

Value 

Objective 

Rationale 

How 

Measured 

Group  -- 

Group  -- 

*Can  be  used  to  compare  relative  importance  of  S&T  Exit  Criteria.  These  are  suggestive,  use  whatever  scale  you 
determine  relevant/needed." 

Priority  qualitative  ranking:  High,  Medium,  Low. ...Or, 

Weight  scores  contribution  importance:  1  =  critical  contribution  to  capability,  0.7  important  contribution,  and 
0.5  if  not  too  important 

Threshold:  The  measurable  value  that  must  be  achieved  for  the  technology  to  advance  to  the  next  stage  of 
development  or  transition  (usually  given  as  a  minimum  or  maximum  value). 

Objective:  The  stretch-goal  value  that  is  desirable,  but  not  essential  (often  viewed  as  the  requirements  trade 
space  if  parameters  such  as  cost  or  schedule  are  more  important  to  the  customer). 


...the  “Conversation”  continues...  When  is  'more  or  less'  better  and  how  do  you  measure  it?" -....and  Why? 
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Step  3:  Generate  Alternatives.  List  ail  the  possible  ways  to  solve  a  problem. 


Worksheet  3.1 
List  Alternatives 


Project  Name: 
Member  Name: . 

Role: . 
Worksheet  Date: 


The  Team  identifies  possible  solutions  (called  Alternatives)  to  satisfy 
each  of  the  S&T  Exit  Criteria.  The  Alternatives  may  be  existing  technologies, 
but  the  team  should  also  explore  novel  approaches  that  may  offer 
enhanced  payoff...  even  if  it  seems  a  higher  risk. 


Alternative  Name 

Description 

...the  “Conversation"  continues. ..What  Alternatives  might  satisfy  the  S&T  Exit  Criteria?  ....and  Why? 
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Worksheet  3.2 
Alternatives 
Complete  Descriptions 

(One  Worksheet  per  Alternative) 


Project  Name: 
Member  Name: . 

Role: . 
Worksheet  Date: 


Describe  the  “attributes"  of  each  Alternative  in  as  great  a  detail  as  possible  as  they  pertain  to 
each  S&T  Exit  Criterion.  Some  form  of  Description/Quad  Chart  presentation  (attached  to 
Worksheet  3.2)  could  make  value  assessment  and  decision  briefing  easier  as  you  go. 


Alternative  Name: 


Description:. 


Any  Considerations: _ 

Estimate  Cost  and  Schedule: _ 

Estimate  Initial  TRL: _  Final  TRL: _ 

Estimate  Initial  MRL: _  Final  MRL: _ 

Estimate  Payoff  to  Maturity  score:  ( Score  the  Alternative  on  the  table  below) 


Mature 

— 

i . i  Low  Payoff 

Alternative 

Picture  or 

Near  Term 

Graphical 

c c 

— 

New  Capability 

Representation 

Immature 

/  Illustration 

Game-Changer 

if  available 

_ 

Payoff 
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Step  4.  Evaluate  Alternatives  (Value  Analysis)  The  Measures  of  Merit 
are  Desirability  ( Worksheet  4.1)  and  Risk  (Worksheet  4.2) 


Worksheet  4.1 
Desirability  for  each 
S&T  Exit  Criteria 
Vs.  each  Alternative 

(One  Worksheet  per  Alternative) 


Project  Name: _ 

Member  Name:  _ 

Role: _ 

Worksheet  Date: 


Certain  Exit  Criteria  (EC)  are  more  important  than  other  Exit  Criteria.  Enter  an  EC  Weight 
value  from  Step  2  to  score  the  relative  importance  of  each  exit  criteria  compared  to  the  other 
exit  criteria. 

Certain  Alternatives  are  more  credible  to  attain  the  Exit  Criteria.  Enter  an  Alternative  (Alt) 
Score  as  to  how  well  the  Alternative  satisfies  each  S&T  Exit  Criterion  based  on  a  team  establish 
scoring  method.  If  any  Alternative  fails  to  meet  one  or  more  S&T  Exit  Criteria,  a  zero  score  is 
entered.  A  zero  score  essentially  fails  that  alternative  (EC  Wt  Score  x  0  =  0),  however,  it  is  still 
recommended  the  Alternative  be  retained  for  the  final  discussion. 

Multiply  EC  Weight  times  the  Alt  Score  (Wt  x  Sc)  for  an  ''EC"  Score.  Add  all  the  EC  scores 
together  for  an  overall  Alternative  Score  (EC1  Score  +  EC2  Score  +  EC3  Score...) 


Alternate  Name: 

EC 

ID 

Exit 

Criteria 

EC 

Wt 

a 

EC  Score 

(WtxSc) 

Rationale 

Alternative  Score  (EC1  Score  +  EC2  Score  +  EC3...) 

DOCUMENT:  The  Team  should  negotiate  one  overall  Alternative  score  for  each  Alternative 
that  is  then  transferred  and  compiled  on  the  Composite  Worksheet  4.3 


The  SE  “Conversation”  continues  "Desirability  is  a  S&T  Exit  Criteria  value  with  room  for  flexibility” 
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Worksheet  4.2  Project  Name: . 

Risk  of  Achieving  Member  Name: . 

each  S&T  Exit  Criterion  Role: . 

Vs.  each  Alternative  Worksheet  Date: 

(One  Worksheet  per  Alternative) 


The  Team  identifies  the  top  Risks  for  each  Alternative  to  meet 
the  threshold  values  for  all  of  the  S&T  Exit  Criteria. 

The  Team  evaluates  the  Risk  of  each  Alternative  against  each  S&T  Exit  Criteria 
and  generates  a  Risk  Score  (Risk  in  this  context  is  the  probability  of  failure  (Pf) 
to  exceed  the  Exit  Criterion  minimum  limits  (Worksheet  2.5)). 

High  Risk  does  not  mean  the  team  should  not  explore  all  possible  Alternatives 
( see  Payoff  to  Maturity  space  on  Worksheet  3.2).  High  Risk  simply  means  the  team 
should  consider  Risk  vs.  Payoff  plus  Mitigation  difficulty.  The  score 
should  also  include  consideration  of  mitigation  difficulty. 


Alternate  Name: 

EC 

ID 

Exit 

Criteria 

Risk 

Description 

Risk 

Score 

Potential  /  Mitigation 

Overall  Risk  Score:  _ 

DOCUMENT:  The  Team  should  negotiate  one  overall  Risk  score  for  each  Alternative.  Each 
Alternative’s  overall  Risk  Score  is  then  transferred  to  the  Composite  Scorecard  -- 
Worksheet  4.3 


...the  “Conversation"  continues. ..What's  the  Probability  of  Failure  (Risk)  to  achieve  each  S&T  Exit  Criterion? 
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Project  Name: 

Worksheet  4.3  Member  Name: . 

Composite  Scorecard  Roie:_ 

Worksheet  Date: 


Transfer  all  Alternative  and  Risk  scores  from  each  Alternative  Worksheets  ( 4.1  and  4.2) 

to  this  Composite  Scorecard 


ALTERNATIVES 

EC 

ID 

Exit  Criteria 

Alternative 

Alternative 

Alternative 

Alternative 

Alternative  Score 

(Worksheet  4.1 ) 

Risk  Score 

(Worksheet  4.2) 

Preferred  Alternative  (s)  / 
Rationale 

Preferred  Alternative (s) 
Decision  Documentation 

The  Composite  Scorecard  is  not  the  final  answer,  but  presents  the  team  with  values  for 
them  to  agree  on  the  Preferred  Alternative(s). 

Analyze  the  scores  by  asking  the  questions: 

1  -  Does  the  ranking  of  the  composite  scores  pass  the  sanity  test,  i.e.,  do  the  values  make  sense? 

2  -  Do  Risks  outweigh  the  scores  for  the  high  scoring  Alternatives? 

Based  on  this  review,  the  team  moves  on  to  the  Action  Plan  for  the  Preferred  Alternative (s). 
Often  in  S&T  it  is  common  to  have  multiple  alternatives  selected  for  initial  development. 


...the  “Conversation"  continues. ..If  a  score  doesn't  make  sense,  ...discuss  why  it  scored  as  it  did? 
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Step  5.  Build  the  Plan:  With  the  Problem  Space  understood  and  the  Solution  Space  fully 
explored,  Advocacy  Briefs  and  Action  Plans  can  be  generated  for  the  Preferred  Alternative  (sj. 


Project  Name: 

Worksheet  5.1  Member  Name:. 

Program  Action  Plan  Role: . 

Worksheet  Date: 


The  Team  should  agree  upon  the  plan  to  advocate  for  approval/implementation  of  the 
proposed  S&T  program  incorporating  the  Preferred  Alternative (s).  Depending  on  the  nature 
of  the  proposed  program,  the  Action  Plan  could  range  from  a  simple  White  Paper  or  Roadmap 
to  a  fully  detailed  advocacy  briefing/package.  Worksheet  5.1  provides  a  possible  outline 
based  on  principles  of  good  program  management. 

Details  necessary  to  complete  this  Action  Plan  include  programmatic  technology 
performance,  costand  schedule  estimates,  a  basic  program  execution  strategy  of  external 
contracts  and  in-house  activities,  as  well  as  the  type  of  funding  (6.1,  6.2,  6.3)  required. 


Program  Action  Plan  Checklist  (possible  outline) 

□  Document  details  of  the  planning  process: 

□  AF  problem 

□  Customer  (s)/owners/end  users 

□  Requirements 

□  S&T  Exit  Criteria 

□  Details  of  each  alternative  solution  considered 

□  Preferred  alternative (s)  selected  (include  rationale) 

□  Document  proposed  program  execution/management  plan 

□  Proposed  S&T  program  schedule  (with  milestones  and  decision  points)  -  may  be 
in  the  form  of  a  technology  roadmap 

□  Required  resources 

□  Proposed  execution  approach  (inhouse,  external  contracts) 

□  Risk  management  approach 

•  Describe  the  recommended  next  steps  in  advocacy  process: 


...the  SE  “Conversation"  never  stops.. .A  well  thought  out  plan  is  easier  to  defend  and  execute. 
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Worksheet  5.2 
SE  Case  Study 


While  the  actions,  activities,  and  experiences  of  applying  the  Systems  Engineering 
Streamlined  Planning  Process  are  still  fresh  in  the  mind  of  the  Core  Team  and  other 
participants,  a  Case  Study  is  a  useful  document  to  capture  the  team’s  discoveries. 

Case  Study 

Description  of  Problem  Space 

Background 
Study  Objectives 

Study  Process 

Description  of  Technical  Effort 
The  Integrated  Product  Team 
Kickoff  Meeting,  date 

Technical  and  Systems  Engineering  Process  Conclusions 

Systems  Engineering 

Additional  Observations 

Lessons  Learned  and  Recommendations 

Appendix  A.  List  of  Acronyms  and  T erms 
Appendix  B.  References 


...the  SE  “Conversation"  never  stops. ..don't  be  surprised  to  find  this  process  wasn't  worth  the  time  invested. 
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